From exim@www1.ietf.org Tue Jul 1 09:24:33 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA24340 for ; Tue, 1 Jul 2003 09:24:32 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h61DO5t09538 for rddp-archive@odin.ietf.org; Tue, 1 Jul 2003 09:24:05 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XL77-0002Tk-6e for rddp-web-archive@optimus.ietf.org; Tue, 01 Jul 2003 09:24:05 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA24091 for ; Tue, 1 Jul 2003 09:23:32 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XL68-0001wu-EV; Tue, 01 Jul 2003 09:23:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XL5p-0001nI-36 for rddp@optimus.ietf.org; Tue, 01 Jul 2003 09:22:45 -0400 Received: from ztxmail03.ztx.compaq.com (ztxmail03.ztx.compaq.com [161.114.1.207]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA23890 for ; Tue, 1 Jul 2003 09:22:41 -0400 (EDT) Received: from cceexg11.americas.cpqcorp.net (unknown [16.110.250.125]) by ztxmail03.ztx.compaq.com (Postfix) with ESMTP id D553ACE15 for ; Tue, 1 Jul 2003 08:22:12 -0500 (CDT) Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Tue, 1 Jul 2003 08:22:12 -0500 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 Date: Tue, 1 Jul 2003 08:22:12 -0500 Message-ID: <4D027986353D1341ADA4F2A4F8A170762AB1E188@cceexc18.americas.cpqcorp.net> Thread-Topic: I-D ACTION:draft-culley-iwarp-mpa-03.txt,.pdf Thread-Index: AcM+/oyuol55rxjyRWS0K1vIEB/AxAA1IoSg From: "Culley, Paul" To: X-OriginalArrivalTime: 01 Jul 2003 13:22:12.0765 (UTC) FILETIME=[C84D14D0:01C33FD3] Content-Transfer-Encoding: quoted-printable Subject: [rddp] I-D ACTION:draft-culley-iwarp-mpa-03.txt,.pdf Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable Of interest to this group... To: IETF-Announce: ;=20 Subject: I-D ACTION:draft-culley-iwarp-mpa-03.txt,.pdf=20 From: Internet-Drafts@ietf.org=20 Date: Tue, 01 Jul 2003 07:22:05 -0400=20 Reply-to: Internet-Drafts@ietf.org=20 Sender: owner-ietf-announce@ietf.org=20 ------------------------------------------------------------------------ -------- A New Internet-Draft is available from the on-line Internet-Drafts directories. Title : Marker PDU Aligned Framing for TCP Specification Author(s) : P. Culley et al. Filename : draft-culley-iwarp-mpa-03.txt,.pdf Pages : 53 Date : 2003-6-30 =09 A framing protocol is defined for TCP that is fully compliant with applicable TCP RFCs and fully interoperable with existing TCP implementations. The framing mechanism is designed to work as an 'adaptation layer' between TCP and the Direct Data Placement [DDP] protocol, preserving the reliable, in-order delivery of TCP, while adding the preservation of higher-level protocol record boundaries that DDP requires. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-culley-iwarp-mpa-03.txt 110539 bytes=20 http://www.ietf.org/internet-drafts/draft-culley-iwarp-mpa-03.pdf 122280 bytes=20 To remove yourself from the IETF Announcement list, send a message to=20 ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-culley-iwarp-mpa-03.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html=20 or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-culley-iwarp-mpa-03.txt". =09 NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. =09 =09 Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 1 11:13:35 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA04096 for ; Tue, 1 Jul 2003 11:13:35 -0400 (EDT) Received: (from exim@localhost) by www1.ietf.org (8.11.6/8.11.6) id h61FD8M10147 for rddp-archive@odin.ietf.org; Tue, 1 Jul 2003 11:13:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XMoe-0002da-Pi for rddp-web-archive@optimus.ietf.org; Tue, 01 Jul 2003 11:13:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA04060 for ; Tue, 1 Jul 2003 11:13:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19XMoc-0005Vr-00 for rddp-web-archive@ietf.org; Tue, 01 Jul 2003 11:13:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19XMoW-0005Vo-00 for rddp-web-archive@ietf.org; Tue, 01 Jul 2003 11:13:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XMoY-0002bz-4L; Tue, 01 Jul 2003 11:13:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XMo8-0002bC-D7 for rddp@optimus.ietf.org; Tue, 01 Jul 2003 11:12:36 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA04035 for ; Tue, 1 Jul 2003 11:12:32 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19XMo5-0005VF-00 for rddp@ietf.org; Tue, 01 Jul 2003 11:12:33 -0400 Received: from fmr05.intel.com ([134.134.136.6] helo=hermes.jf.intel.com) by ietf-mx with esmtp (Exim 4.12) id 19XMnu-0005Ud-00 for rddp@ietf.org; Tue, 01 Jul 2003 11:12:22 -0400 Received: from petasus.jf.intel.com (petasus.jf.intel.com [10.7.209.6]) by hermes.jf.intel.com (8.11.6p2/8.11.6/d: outer.mc,v 1.66 2003/05/22 21:17:36 rfjohns1 Exp $) with ESMTP id h61F9Bd03273 for ; Tue, 1 Jul 2003 15:09:11 GMT Received: from orsmsxvs040.jf.intel.com (orsmsxvs040.jf.intel.com [192.168.65.206]) by petasus.jf.intel.com (8.11.6p2/8.11.6/d: inner.mc,v 1.35 2003/05/22 21:18:01 rfjohns1 Exp $) with SMTP id h61F6XK01113 for ; Tue, 1 Jul 2003 15:06:34 GMT Received: from orsmsx331.amr.corp.intel.com ([192.168.65.56]) by orsmsxvs040.jf.intel.com (NAVGW 2.5.2.11) with SMTP id M2003070108221914694 ; Tue, 01 Jul 2003 08:22:19 -0700 Received: from orsmsx407.amr.corp.intel.com ([192.168.65.50]) by orsmsx331.amr.corp.intel.com with Microsoft SMTPSVC(5.0.2195.5329); Tue, 1 Jul 2003 08:11:07 -0700 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 Subject: RE: [rddp] Security Draft issues - STag / PDs. Date: Tue, 1 Jul 2003 08:11:07 -0700 Message-ID: <42050DF556283A4D977B111EB706320807417F@orsmsx407.jf.intel.com> Thread-Topic: [rddp] Security Draft issues - STag / PDs. Thread-Index: AcM/c8jEsd1T5tFNQA2A6pSIg5SfKAAbIOtg From: "Deleganes, Ellen M" To: "Caitlin Bestler" , Cc: X-OriginalArrivalTime: 01 Jul 2003 15:11:07.0743 (UTC) FILETIME=[FF7396F0:01C33FE2] Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable My comments are interspersed. Ellen Deleganes -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com]=20 Sent: Monday, June 30, 2003 6:52 PM To: Black_David@emc.com Cc: rddp@ietf.org Subject: Re: [rddp] Security Draft issues - STag / PDs. On Monday, June 30, 2003, Black_David@emc.com wrote: >Caitlin Bestler wrote: > >> My reading of the DDP spec is that setting the Protection Domain >> is *the* way to control the scope of an STag. > >I don't think so. Section 10.3 of the DDP draft describes both >the Protection Domain and stream association mechanisms and then says: > > DDP MUST support=20 > Protection Domain association and DDP stream association mechanisms=20 > for associating an STag and a DDP Stream. =20 > >IMHO, that implies that the functionality of using stream association >within a Protection Domain (as envisioned by the security draft) is >REQUIRED. IIRC, this topic was discussed in Atlanta, but I'm not >going to object too strenuously to reopening it in the context of >the security draft; keep in mind that both models were felt to be >useful at the time. Funny, I read that section as describing two different interfaces to the same mechanism. Specifically: >10.3 Association of an STag and a DDP Stream=20 > > There are several mechanisms for associating an STag and a DDP=20 > Stream. Two reasonable mechanisms for this association are a=20 > Protection Domain (PD) association and a DDP Stream association.=20 > > Under the Protection Domain (PD) association, a unique Protection=20 > Domain Identifier (PD ID) is created and used locally to associate=20 > an STag with a set of DDP Streams. Under this mechanism, the use of=20 > the STag is only permitted on the DDP Streams that have the same PD=20 > ID as the STag. For an incoming DDP Segment of a Tagged DDP Message=20 > on a DDP Stream, if the PD ID of the DDP Stream is not the same as=20 > the PD ID of the STag targeted by the Tagged DDP Message, then the=20 > DDP Segment is not placed and the DDP layer MUST surface a local=20 > error to the ULP. Note that the PD ID is locally defined, and cannot > be directly manipulated by the Remote Peer.=20 >=20 > Under the DDP Stream association, a DDP Stream is identified locally > by a unique DDP Stream identifier (ID). > Which is really the same thing as a Protection Domain Identifier, except that the Consumer does not have to generate it. > An STag is associated with a=20 > DDP Stream by using a DDP Stream ID. In this case, for an incoming=20 > DDP Segment of a Tagged DDP Message on a DDP Stream, if the DDP=20 > Stream ID of the DDP Stream is not the same as the DDP Stream ID of=20 > the STag targeted by the Tagged DDP Message, then the DDP Segment is > not placed and the DDP layer MUST surface a local error to the ULP.=20 > Note that the DDP Stream ID is locally defined, and cannot be=20 > directly manipulated by the Remote Peer.=20 > So that in either case the "Protection Identifier" of the DDP Stream is compared with the STag's "Protection Identifier". > A ULP SHOULD associate an STag and a DDP Stream. DDP MUST support=20 > Protection Domain association and DDP Stream association mechanisms=20 > for associating an STag and a DDP Stream. =20 And *nowhere* does it imply that a single DDP Stream needs to be associated with *two* "Protection Identifiers", or that an STag ever has to be associated with *two* "Protection Identifiers". In any event, I interpreted the security-draft text as specifically implying that you could have both methods active on the *same* stream or the same STag. Personally, I'm working on a software implementation, so a quick OR that isn't going to make that much difference in performance, especially if optimizated tso that the matching id more often the first one. But if that was meant to imply checking two values then it clearly needs to be reworded. But my recomendation is for a single check. It's simple. It's quick, and its adequate. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 2 10:51:31 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA25706 for ; Wed, 2 Jul 2003 10:51:31 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Xiwq-0005P5-Ua for rddp-archive@odin.ietf.org; Wed, 02 Jul 2003 10:51:05 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h62Ep4F4020767 for rddp-archive@odin.ietf.org; Wed, 2 Jul 2003 10:51:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Xiwq-0005Os-Rk for rddp-web-archive@optimus.ietf.org; Wed, 02 Jul 2003 10:51:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA25697 for ; Wed, 2 Jul 2003 10:51:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Xiwo-0001UX-00 for rddp-web-archive@ietf.org; Wed, 02 Jul 2003 10:51:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Xiwn-0001UT-00 for rddp-web-archive@ietf.org; Wed, 02 Jul 2003 10:51:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Xiwn-0005OX-Mj; Wed, 02 Jul 2003 10:51:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XiwU-0005O9-4n for rddp@optimus.ietf.org; Wed, 02 Jul 2003 10:50:42 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA25654 for ; Wed, 2 Jul 2003 10:50:37 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19XiwR-0001TT-00 for rddp@ietf.org; Wed, 02 Jul 2003 10:50:39 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19XiwQ-0001T2-00 for rddp@ietf.org; Wed, 02 Jul 2003 10:50:38 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id A52CD245774; Wed, 2 Jul 2003 10:50:04 -0400 (EDT) Date: Wed, 2 Jul 2003 09:50:03 -0500 From: Caitlin Bestler Subject: RE: [rddp] Security Draft issues - STag / PDs. To: "Deleganes, Ellen M" , rddp@ietf.org X-Priority: 3 In-Reply-To: <42050DF556283A4D977B111EB706320807417F@orsmsx407.jf.intel.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit If I'm following what you are saying, a minimal compliant implementation would work like this: All DDP Streams have a unique DDP Stream Identifier. This also serves as their "Protection ID" by default. If the ULP assigns a Protection Domain to a DDP Stream then the Protection Domain ID becomes the Protection ID. If the local interface allows a Protection Domain to be *removed* (something that has not traditionally been allowed) then the "Protection ID" would revert to the DDP Stream Identifier. An STag must have a Protection ID associated with it, but it can be either form. An incoming STag is checked by one of the following algorithms: simple compare: STag.Protection ID == DDP Stream.Protection ID scoped lookup: exists(Protection ID + STag) Further, a local interface MAY allow multiple Protection IDs to be associated with a DDP Stream. For example it could allow the DDP Stream Identifier to remain valid even after a Protection Domain was assigned, or it could even allow multiple Protection Domains to be assigned. Extra local interface capabilities have no impact on the wire protocol. The end-to-end semantics remain that the local interface must be used to enable a STag on the set of DDP Streams that the ULP says it may be used upon. The other end only cares about the result. If this is correct, then the DDP draft might want to clarify the relationship between the two mechanisms. I would also say that if the above understanding is correct, that the reference in the security-draft is incorrect. It would be more appropriate to note what the required capability is while noting that a local interface MAY offer the ULP more flexibility. It is only a security concern to the extent that offering the ULP more rope could allow it to hang itself, but I think we can all agree that the standard capabilities already allow a ULP sufficient rope to hang itself if it *wants* to be careless. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 2 12:46:31 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA00854 for ; Wed, 2 Jul 2003 12:46:31 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Xkk7-0006H2-Ib for rddp-archive@odin.ietf.org; Wed, 02 Jul 2003 12:46:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h62Gk3rK024112 for rddp-archive@odin.ietf.org; Wed, 2 Jul 2003 12:46:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Xkk7-0006Gp-FV for rddp-web-archive@optimus.ietf.org; Wed, 02 Jul 2003 12:46:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA00840 for ; Wed, 2 Jul 2003 12:45:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Xkk5-0003qG-00 for rddp-web-archive@ietf.org; Wed, 02 Jul 2003 12:46:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Xkk5-0003qD-00 for rddp-web-archive@ietf.org; Wed, 02 Jul 2003 12:46:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Xkk6-0006GV-5O; Wed, 02 Jul 2003 12:46:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XkjZ-0006G7-W9 for rddp@optimus.ietf.org; Wed, 02 Jul 2003 12:45:32 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA00831 for ; Wed, 2 Jul 2003 12:45:20 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19XkjT-0003py-00 for rddp@ietf.org; Wed, 02 Jul 2003 12:45:23 -0400 Received: from ztxmail05.ztx.compaq.com ([161.114.1.209]) by ietf-mx with esmtp (Exim 4.12) id 19XkjS-0003pp-00 for rddp@ietf.org; Wed, 02 Jul 2003 12:45:22 -0400 Received: from cceexg13.americas.cpqcorp.net (cceexg13.americas.cpqcorp.net [16.110.250.119]) by ztxmail05.ztx.compaq.com (Postfix) with ESMTP id 73729CAE8; Wed, 2 Jul 2003 11:44:52 -0500 (CDT) Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg13.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Wed, 2 Jul 2003 11:44:52 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="US-ASCII" Content-Transfer-Encoding: quoted-printable Subject: RE: [rddp] Security Draft issues - STag / PDs. Date: Wed, 2 Jul 2003 11:44:51 -0500 Message-ID: <4D027986353D1341ADA4F2A4F8A170762AB1E1E8@cceexc18.americas.cpqcorp.net> Thread-Topic: [rddp] Security Draft issues - STag / PDs. Thread-Index: AcNAqYigGkv+LteCTgyqjk3KXFuzXwADPqsw From: "Culley, Paul" To: "Caitlin Bestler" , X-OriginalArrivalTime: 02 Jul 2003 16:44:52.0131 (UTC) FILETIME=[4242D730:01C340B9] Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable I think the issue would be clarified by taking some examples from the "verbs" draft (draft-hilland-rddp-verbs-00). That draft says that each DDP stream (identified with a QP ID) is also always assigned a "Protection Domain" or PD on request of the ULP. Some trusted component makes sure that the ULP is allowed to use that PD, either by assigning a new one, or otherwise checking that the ULP is allowed to reuse an already present one (possibly via a process ID). A QP is only in a single PD, but many QPs may be assigned to a PD, allowing certain cross QP remote access possibilities. When an STag is allocated, it is initially assigned to a PD as well on request of the ULP; the ULP is only allowed to assign a PD that it "owns" (this is to be enforced by a trusted and protected component, typically a module associated with the OS). When the STag is "registered" by the ULP to allow remote access, it may continue to use the PD (allowing access by any QP with the same PD), or be further "bound" to a specific QP (which thereafter limits access to the single QP referenced DDP stream). The registration procedure also enforces that the ULP is accessing memory it owns (again, typically done with a trusted component, associated with the OS). When the STag is invalidated, it reverts to PD level protection, allowing binding on any QP in the same PD the next time it is "Bound". We should perhaps adjust our language in the security draft to tighten our requirements so that this type of confusion does not occur. Paul R. Culley HP Senior Fellow 281-514-5543 > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com]=20 > Sent: Wednesday, July 02, 2003 9:50 AM > To: Deleganes, Ellen M; rddp@ietf.org > Subject: RE: [rddp] Security Draft issues - STag / PDs. >=20 >=20 > If I'm following what you are saying, a minimal compliant=20 > implementation would work like this: >=20 >=20 > All DDP Streams have a unique DDP Stream Identifier. > =20 > This also serves as their "Protection ID" by default. > =20 > If the ULP assigns a Protection Domain to a DDP Stream > then the Protection Domain ID becomes the Protection ID. > =20 > If the local interface allows a Protection Domain to > be *removed* (something that has not traditionally > been allowed) then the "Protection ID" would revert > to the DDP Stream Identifier. > =20 > An STag must have a Protection ID associated with it, > but it can be either form. > =20 > An incoming STag is checked by one of the following > algorithms: > =20 > simple compare: STag.Protection ID =3D=3D DDP = Stream.Protection ID > scoped lookup: exists(Protection ID + STag) > =20 > Further, a local interface MAY allow multiple Protection IDs > to be associated with a DDP Stream. For example it could > allow the DDP Stream Identifier to remain valid even after > a Protection Domain was assigned, or it could even allow=20 > multiple Protection Domains to be assigned. >=20 > Extra local interface capabilities have no impact on the > wire protocol. The end-to-end semantics remain that the > local interface must be used to enable a STag on the set > of DDP Streams that the ULP says it may be used upon. > The other end only cares about the result. >=20 > If this is correct, then the DDP draft might want to clarify=20 > the relationship between the two mechanisms. I would also say=20 > that if the above understanding is correct, that the=20 > reference in the security-draft is incorrect. It would be=20 > more appropriate to note what the required capability is=20 > while noting that a local interface MAY offer the ULP more=20 > flexibility. It is only a security concern to the extent that=20 > offering the ULP more rope could allow it to hang itself, but=20 > I think we can all agree that the standard capabilities=20 > already allow a ULP sufficient rope to hang itself if it=20 > *wants* to be careless. >=20 > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp >=20 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 2 13:56:33 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04136 for ; Wed, 2 Jul 2003 13:56:33 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Xlpt-0002Lc-Kf for rddp-archive@odin.ietf.org; Wed, 02 Jul 2003 13:56:05 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h62Hu5B3009020 for rddp-archive@odin.ietf.org; Wed, 2 Jul 2003 13:56:05 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Xlpt-0002LP-HS for rddp-web-archive@optimus.ietf.org; Wed, 02 Jul 2003 13:56:05 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04116 for ; Wed, 2 Jul 2003 13:56:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Xlpr-0004ze-00 for rddp-web-archive@ietf.org; Wed, 02 Jul 2003 13:56:03 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Xlpq-0004zb-00 for rddp-web-archive@ietf.org; Wed, 02 Jul 2003 13:56:02 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Xlpr-0002Ko-2z; Wed, 02 Jul 2003 13:56:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Xlpk-0002KS-DB for rddp@optimus.ietf.org; Wed, 02 Jul 2003 13:55:56 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04113 for ; Wed, 2 Jul 2003 13:55:53 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Xlpi-0004zX-00 for rddp@ietf.org; Wed, 02 Jul 2003 13:55:54 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19Xlph-0004zE-00 for rddp@ietf.org; Wed, 02 Jul 2003 13:55:53 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id BAB43245475; Wed, 2 Jul 2003 13:55:21 -0400 (EDT) Date: Wed, 2 Jul 2003 12:55:20 -0500 From: Caitlin Bestler Subject: RE: [rddp] Security Draft issues - STag / PDs. To: "Culley, Paul" Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <4D027986353D1341ADA4F2A4F8A170762AB1E1E8@cceexc18.americas.cpqcorp.net> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Wednesday, July 2, 2003, paul.culley@hp.com wrote: >I think the issue would be clarified by taking some examples from the >"verbs" draft (draft-hilland-rddp-verbs-00). > Well they would be, but they shouldn't be. The verbs draft, by its own description, covers *one* style of local interface that *may* be used. Explanations on the definition of an STag, discussions of applicability and security should all be valid across the full set of possible local interfaces. >That draft says that each DDP stream (identified with a QP ID) >is also always assigned a "Protection Domain" or PD on request >of the ULP. Some trusted component makes sure that the ULP is >allowed to use that PD, either by assigning a new one, or >otherwise checking that the ULP is allowed to reuse an already >present one (possibly via a process ID). > >A QP is only in a single PD, but many QPs may be assigned to a PD, >allowing certain cross QP remote access possibilities. > >When an STag is allocated, it is initially assigned to a PD as well on >request of the ULP; the ULP is only allowed to assign a PD that it >"owns" (this is to be enforced by a trusted and protected component, >typically a module associated with the OS). > >When the STag is "registered" by the ULP to allow remote >access, it may continue to use the PD (allowing access by any >QP with the same PD), or be further "bound" to a specific QP >(which thereafter limits access to the single QP referenced DDP >stream). The registration procedure also enforces that the ULP >is accessing memory it owns (again, typically done with a >trusted component, associated with the OS). When the STag is >invalidated, it reverts to PD level protection, allowing >binding on any QP in the same PD the next time it is "Bound". > The above discussion mixes internal concepts with external ones. "Protection Domain" is being used as both a "local memory access authorization mechanism" and with its external meaning. I prefer to work from the external definition only. Hence the ULP creates one or more Protection Domains, enabling that Protection Domain with some subset of the memory access rights that it itself has. Endpoints and STags can be created/associated with Protection Domains. With these externally-oriented definitions, certain things become obvious: -- you should never place multiple STags or Endpoints in the same Protection Domain unless there is a ULP need to do so. The security draft seems to assume that a Protection Domain is matches the scope of the user process by default and provides complex advice on when to narrow it. With the external definition the advice is simple, don't make a Protection Domain any wider than the ULP requires. -- The "stream identifier" is really just a degenerate subset of a default Protection Domain encompassing a single Endpoint and STags that can only be used on that Endpoint. Now the local interface definitely should have a "Memory Regitration Domain" or "Local Memory Region" that can encompass multiple Remote Protection Domains. But that is a *local interface* optimization, it allows the ULP to register memory *locally* once and enable it for multiple Protection Domains. It simplifies coding logic, and makes it easier for the RDMA stack to minimize address mapping resources. But is it really relevant to the protocol, applicability or security? I remain uncertain as to whether this is an "optional" feature that is not being adequately described, or mere fumbling with terminology because of presumptions about internal architectures. I thought it had been settled that a remotely visible STag (the only type that exists in the protocol) is defined by *one* "Protection ID" and that *all* STags valid on the same DDP Stream would have to have the *same* "Protection ID". Early language that implied that the local interface would simply enable valid STag/Stream pairings was specifically *rejected* because it implied that the RNIC would have to search a list in order to validate, rather than perform a single comparison. If we're not doing *one*, we should revert to the simplicity of "many". This "one except when maybe it might be two" is confusing. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 2 14:36:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA05666 for ; Wed, 2 Jul 2003 14:36:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XmSZ-0003yg-5f for rddp-archive@odin.ietf.org; Wed, 02 Jul 2003 14:36:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h62Ia3Z5015291 for rddp-archive@odin.ietf.org; Wed, 2 Jul 2003 14:36:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XmSZ-0003yY-2E for rddp-web-archive@optimus.ietf.org; Wed, 02 Jul 2003 14:36:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA05647 for ; Wed, 2 Jul 2003 14:35:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19XmSW-0005Zy-00 for rddp-web-archive@ietf.org; Wed, 02 Jul 2003 14:36:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19XmSV-0005Zv-00 for rddp-web-archive@ietf.org; Wed, 02 Jul 2003 14:35:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XmSX-0003y5-5g; Wed, 02 Jul 2003 14:36:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XmSB-0003xt-3q for rddp@optimus.ietf.org; Wed, 02 Jul 2003 14:35:39 -0400 Received: from caduceus.jf.intel.com (fmr06.intel.com [134.134.136.7]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA05637 for ; Wed, 2 Jul 2003 14:35:35 -0400 (EDT) Received: from petasus.jf.intel.com (petasus.jf.intel.com [10.7.209.6]) by caduceus.jf.intel.com (8.11.6p2/8.11.6/d: outer.mc,v 1.66 2003/05/22 21:17:36 rfjohns1 Exp $) with ESMTP id h62IOUh22437 for ; Wed, 2 Jul 2003 18:24:30 GMT Received: from orsmsxvs040.jf.intel.com (orsmsxvs040.jf.intel.com [192.168.65.206]) by petasus.jf.intel.com (8.11.6p2/8.11.6/d: inner.mc,v 1.35 2003/05/22 21:18:01 rfjohns1 Exp $) with SMTP id h62IPNb13607 for ; Wed, 2 Jul 2003 18:25:23 GMT Received: from orsmsx332.amr.corp.intel.com ([192.168.65.60]) by orsmsxvs040.jf.intel.com (NAVGW 2.5.2.11) with SMTP id M2003070211411303282 ; Wed, 02 Jul 2003 11:41:13 -0700 Received: from orsmsx407.amr.corp.intel.com ([192.168.65.50]) by orsmsx332.amr.corp.intel.com with Microsoft SMTPSVC(5.0.2195.5329); Wed, 2 Jul 2003 11:28:55 -0700 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 Subject: RE: [rddp] Security Draft issues - STag / PDs. Date: Wed, 2 Jul 2003 11:28:55 -0700 Message-ID: <42050DF556283A4D977B111EB7063208074188@orsmsx407.jf.intel.com> Thread-Topic: [rddp] Security Draft issues - STag / PDs. Thread-Index: AcNAw0DWjcYUYlBKRDC9N5N9qPkw+AAAnTRw From: "Deleganes, Ellen M" To: "Caitlin Bestler" , "Culley, Paul" Cc: X-OriginalArrivalTime: 02 Jul 2003 18:28:55.0520 (UTC) FILETIME=[CB9C6600:01C340C7] Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable My comments are embedded. Ellen -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com]=20 Sent: Wednesday, July 02, 2003 10:55 AM To: Culley, Paul Cc: rddp@ietf.org Subject: RE: [rddp] Security Draft issues - STag / PDs. On Wednesday, July 2, 2003, paul.culley@hp.com wrote: >I think the issue would be clarified by taking some examples from the >"verbs" draft (draft-hilland-rddp-verbs-00). > Well they would be, but they shouldn't be. The verbs draft, by its own description, covers *one* style of local interface that *may* be used. Explanations on the definition of an STag, discussions of applicability and security should all be valid across the full set of possible local interfaces. >That draft says that each DDP stream (identified with a QP ID) >is also always assigned a "Protection Domain" or PD on request >of the ULP. Some trusted component makes sure that the ULP is >allowed to use that PD, either by assigning a new one, or >otherwise checking that the ULP is allowed to reuse an already >present one (possibly via a process ID). > >A QP is only in a single PD, but many QPs may be assigned to a PD, >allowing certain cross QP remote access possibilities. > >When an STag is allocated, it is initially assigned to a PD as well on >request of the ULP; the ULP is only allowed to assign a PD that it >"owns" (this is to be enforced by a trusted and protected component, >typically a module associated with the OS). > >When the STag is "registered" by the ULP to allow remote >access, it may continue to use the PD (allowing access by any >QP with the same PD), or be further "bound" to a specific QP >(which thereafter limits access to the single QP referenced DDP >stream). The registration procedure also enforces that the ULP >is accessing memory it owns (again, typically done with a >trusted component, associated with the OS). When the STag is >invalidated, it reverts to PD level protection, allowing >binding on any QP in the same PD the next time it is "Bound". > The above discussion mixes internal concepts with external ones. "Protection Domain" is being used as both a "local memory access authorization mechanism" and with its external meaning. I prefer to work from the external definition only. Hence the ULP creates one or more Protection Domains, enabling that Protection Domain with some subset of the memory access rights that it itself has. Endpoints and STags can be created/associated with Protection Domains. With these externally-oriented definitions, certain things become obvious: -- you should never place multiple STags or Endpoints in the same Protection Domain unless there is a ULP need to do so. The security draft seems to assume that a Protection Domain is matches the scope of the user process by default and provides complex advice on when to narrow it. With the external definition the advice is simple, don't make a Protection Domain any wider than the ULP requires. -- The "stream identifier" is really just a degenerate subset of a default Protection Domain encompassing a single Endpoint and STags=20 that can only be used on that Endpoint. =20 Now the local interface definitely should have a "Memory Regitration Domain" or "Local Memory Region" that can encompass multiple Remote Protection Domains. But that is a *local interface* optimization, it allows the ULP to register memory *locally* once and enable it for multiple Protection Domains. It simplifies coding logic, and makes it easier for the RDMA stack to minimize address mapping resources. But is it really relevant to the protocol, applicability or security? I remain uncertain as to whether this is an "optional" feature that is not being adequately described, or mere fumbling with terminology because of presumptions about internal architectures. I thought it had been settled that a remotely visible STag (the only type that exists in the protocol) is defined by *one* "Protection ID" and that *all* STags valid on the same DDP Stream would have to have the *same* "Protection ID". Early language that implied that the local interface would simply enable valid STag/Stream pairings was specifically *rejected* because it implied that the RNIC would have to search a list in order to validate, rather than perform a single comparison. If we're not doing *one*, we should revert to the simplicity of "many". This "one except when maybe it might be two" is confusing. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 2 14:48:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA05938 for ; Wed, 2 Jul 2003 14:48:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XmeA-0004U9-5i for rddp-archive@odin.ietf.org; Wed, 02 Jul 2003 14:48:02 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h62Im2rE017240 for rddp-archive@odin.ietf.org; Wed, 2 Jul 2003 14:48:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XmeA-0004Tz-0x for rddp-web-archive@optimus.ietf.org; Wed, 02 Jul 2003 14:48:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA05921 for ; Wed, 2 Jul 2003 14:47:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Xme7-0005gu-00 for rddp-web-archive@ietf.org; Wed, 02 Jul 2003 14:47:59 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Xme6-0005gr-00 for rddp-web-archive@ietf.org; Wed, 02 Jul 2003 14:47:58 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Xme8-0004Tf-Fq; Wed, 02 Jul 2003 14:48:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19XmdU-0004TH-OL for rddp@optimus.ietf.org; Wed, 02 Jul 2003 14:47:20 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA05905 for ; Wed, 2 Jul 2003 14:47:17 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19XmdS-0005gT-00 for rddp@ietf.org; Wed, 02 Jul 2003 14:47:18 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19XmdR-0005g1-00 for rddp@ietf.org; Wed, 02 Jul 2003 14:47:17 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id D7CE624564C; Wed, 2 Jul 2003 14:46:46 -0400 (EDT) Date: Wed, 2 Jul 2003 13:46:45 -0500 From: Caitlin Bestler Subject: RE: [rddp] Security Draft issues - STag / PDs. To: "Deleganes, Ellen M" Cc: "Culley, Paul" , rddp@ietf.org X-Priority: 3 In-Reply-To: <42050DF556283A4D977B111EB7063208074188@orsmsx407.jf.intel.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Wednesday, July 2, 2003, ellen.m.deleganes@intel.com wrote: >associated with more than one Protection Domain Identifier. I can see >where one might want to associate multiple DDP Streams with the same >Protection Domain Identifier, but not vice versa. If there is a >programming model that requires this, please explain. Otherwise, it >complicates implementations much more than having the STag valid on a >Protection Domain Identifier OR the DDP Stream identifier.> Allowing any set of Protection Domains to be valid for a DDP Stream is the most general model. I can't really think of a strong reason for doing it. It came up because the minimal statement that the ULP simply enables STags with DDP Streams would have required it. The whole point of the Protection Domain was to allow a *single* check while allowing an STag to be valid on *multiple* DDP Streams. Now the next issue is whether you are saying "having the STag be valid on a Protection Domain Id XOR the DDP Stream ID" or if you are specifying IOR. With XOR, there is one check and the value being checked against might be a Protection Domain ID or a Stream ID, you really have a single tag of a parent type. As I have indicated previously, all you have done is create a special name for a default Protection Domain that the consumer does not have to explicitly create. With IOR, you have a dual check. Now if you have mistakenly set the Protection Domain to be the scope of the application program rather than to the set of DDP Streams associated with a single ULP session then the IOR might look like a neat idea. But if you narror the Protection Domain to the ULP required scope to begin with, I do not see why you would want to further narrow it for *some* STags. Multiple Stream sessions are somewhat of a specialized application to begin with. Multiple Stream sessions that are only kind-of-related to each other for some STags strikes me as horrendously specialized -- and certainly not worth doubling the complexity of the check for *all* STags. Now it may be possible to minimize that performance penalty by "zoning" the Stream IDs and the Protection Domain IDs, but it all strikes me as unnessary complexity. An STag belonging to one Protectin Scope, which is either a Protection Domain (i.e., a set of DDP Streams), or a single DDP Stream, is very simple, to understand, to implement and to use. It also matches existing RDMA-aware APIs. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 3 13:03:35 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA24870 for ; Thu, 3 Jul 2003 13:03:35 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Y7U8-00089K-B9 for rddp-archive@odin.ietf.org; Thu, 03 Jul 2003 13:03:06 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h63H34T6031322 for rddp-archive@odin.ietf.org; Thu, 3 Jul 2003 13:03:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Y7U8-000897-7d for rddp-web-archive@optimus.ietf.org; Thu, 03 Jul 2003 13:03:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA24816 for ; Thu, 3 Jul 2003 13:03:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Y7U6-0003FW-00 for rddp-web-archive@ietf.org; Thu, 03 Jul 2003 13:03:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Y7U5-0003FS-00 for rddp-web-archive@ietf.org; Thu, 03 Jul 2003 13:03:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Y7U5-000882-Dp; Thu, 03 Jul 2003 13:03:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Y7Tx-00087Y-0O for rddp@optimus.ietf.org; Thu, 03 Jul 2003 13:02:53 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA24798 for ; Thu, 3 Jul 2003 13:02:51 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Y7Tv-0003Ev-00 for rddp@ietf.org; Thu, 03 Jul 2003 13:02:51 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19Y7Tu-0003Ef-00 for rddp@ietf.org; Thu, 03 Jul 2003 13:02:50 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 2A3B224573A; Thu, 3 Jul 2003 13:02:15 -0400 (EDT) Date: Thu, 3 Jul 2003 12:02:13 -0500 From: Caitlin Bestler Subject: RE: [rddp] Security Draft issues - STag / PDs. To: "Deleganes, Ellen M" Cc: "Culley, Paul" , rddp@ietf.org X-Priority: 3 In-Reply-To: <42050DF556283A4D977B111EB7063208074188@orsmsx407.jf.intel.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Wednesday, July 2, 2003, ellen.m.deleganes@intel.com wrote: >discussions of applicability will never be done. The full set of >possible local interfaces is probably infinite. However, it may be a >reasonable request to expand the discussions of applicability and >security to a larger set of "reasonable" (useful) local interfaces.> Actually it is very easy. You simply confine your discussion of Protection Domains to the degree that the end-to-end semantics require them to exist. Specifically: A Protection Domain is a container for multiple DDP Streams and STags. All STags within the Protection Domain are valid upon all DDP Streams within the Protection Domain. The Protection Domain SHOULD include no more DDP Streams than are required by the ULP. An application SHOULD NOT include unrelated DDP Streams merely because there is a local desire to share a buffer pool. The local interface SHOULD NOT place pressure on the ULP to merge unrelated Protection Domains into a single Protection Domain merely to reduce RNIC resource consumption. [i.e., there needs to be someone to share mapping tables across multiple Protection Domains when referencing the same shared buffer region]. The local interface MUST allow the consumer to associate an STag with remote read and/or write permission for any contiguous subset of the consumer's address space. The local interface MAY allow the consumre to associate an STag with other sets of memory locations that might not be the consumer's current memory map, might be contiguous, etc. However, the local interface MUST NOT allow the consumer to grant the RNIC any memory access privileges that the consumer did not already have. The local interface MAY require the consumer to preregister memory to be used for reception of untagged messages. The local interface MAY even choose to call the handle for this type an "STag", if it really wants to confuse developers. But it really SHOULD call it something like an LMR Context or an L Key, so that everyone remembers that it is not the same thing as a handle for a remotely accessible memory window (that is called an STag, or an RMR Context / R Key). _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 3 14:18:32 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA29698 for ; Thu, 3 Jul 2003 14:18:32 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Y8ei-00033O-6b for rddp-archive@odin.ietf.org; Thu, 03 Jul 2003 14:18:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h63II4lG011732 for rddp-archive@odin.ietf.org; Thu, 3 Jul 2003 14:18:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Y8ei-000339-3a for rddp-web-archive@optimus.ietf.org; Thu, 03 Jul 2003 14:18:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA29682 for ; Thu, 3 Jul 2003 14:18:01 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Y8ef-0005xI-00 for rddp-web-archive@ietf.org; Thu, 03 Jul 2003 14:18:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Y8ef-0005xF-00 for rddp-web-archive@ietf.org; Thu, 03 Jul 2003 14:18:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Y8ef-00032p-Cj; Thu, 03 Jul 2003 14:18:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Y8dz-00032J-8E for rddp@optimus.ietf.org; Thu, 03 Jul 2003 14:17:19 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA29669 for ; Thu, 3 Jul 2003 14:17:17 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Y8dw-0005wm-00 for rddp@ietf.org; Thu, 03 Jul 2003 14:17:16 -0400 Received: from palrel11.hp.com ([156.153.255.246]) by ietf-mx with esmtp (Exim 4.12) id 19Y8dv-0005wi-00 for rddp@ietf.org; Thu, 03 Jul 2003 14:17:16 -0400 Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel11.hp.com (Postfix) with ESMTP id 08A931C02354 for ; Thu, 3 Jul 2003 11:17:12 -0700 (PDT) Received: from mk731915.cup.hp.com ([15.244.169.211]) by esmail.cup.hp.com (8.11.1/8.8.6) with ESMTP id h63IC9S21190 for ; Thu, 3 Jul 2003 11:12:09 -0700 (PDT) Message-Id: <5.1.0.14.2.20030703111103.0588f540@15.13.130.73> X-Sender: krause@15.13.130.73 X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Thu, 03 Jul 2003 11:17:09 -0700 To: rddp@ietf.org From: Michael Krause Subject: RE: [rddp] Security Draft issues - STag / PDs. In-Reply-To: References: <42050DF556283A4D977B111EB7063208074188@orsmsx407.jf.intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 12:02 PM 7/3/2003 -0500, Caitlin Bestler wrote: >On Wednesday, July 2, 2003, ellen.m.deleganes@intel.com wrote: > > > >discussions of applicability will never be done. The full set of > >possible local interfaces is probably infinite. However, it may be a > >reasonable request to expand the discussions of applicability and > >security to a larger set of "reasonable" (useful) local interfaces.> > >Actually it is very easy. You simply confine your discussion of >Protection Domains to the degree that the end-to-end semantics >require them to exist. > >Specifically: > >A Protection Domain is a container for multiple DDP Streams >and STags. All STags within the Protection Domain are valid >upon all DDP Streams within the Protection Domain. Agreed. >The Protection Domain SHOULD include no more DDP Streams than >are required by the ULP. An application SHOULD NOT include >unrelated DDP Streams merely because there is a local desire >to share a buffer pool. Conceptually true but not possible to enforce and in some cases, the desire to share a buffer region is quite strong and valid for the ULP in question. I do not believe these should be included in any draft as strongly as you have shown above. >The local interface SHOULD NOT place pressure on the ULP >to merge unrelated Protection Domains into a single Protection >Domain merely to reduce RNIC resource consumption. [i.e., there >needs to be someone to share mapping tables across multiple >Protection Domains when referencing the same shared buffer >region]. Need to separate the v-to-p mappings which may be shared with the access controls which are the STag / PD which may or may not be shared. Again, I don't see how what you propose is enforced thus it would be more of a recommendation than a SHOULD NOT. >The local interface MUST allow the consumer to associate >an STag with remote read and/or write permission for any >contiguous subset of the consumer's address space. This would need to be constrained a bit. It may be for a subset of the contiguous range - not necessarily all as one may not be able to pin-n-map the entire range for a given operation. >The local interface MAY allow the consumre to associate >an STag with other sets of memory locations that might not >be the consumer's current memory map, might be contiguous, etc. >However, the local interface MUST NOT allow the consumer to >grant the RNIC any memory access privileges that the consumer >did not already have. > >The local interface MAY require the consumer to preregister >memory to be used for reception of untagged messages. The >local interface MAY even choose to call the handle for this >type an "STag", if it really wants to confuse developers. >But it really SHOULD call it something like an LMR Context >or an L Key, so that everyone remembers that it is not the >same thing as a handle for a remotely accessible memory window >(that is called an STag, or an RMR Context / R Key). Registration is really about v-to-p mappings. Doing it early prior to usage amortizes cost and thus reduces performance penalties. I don't see a reason to change terms between a local and a remote STag. The functionality is the same, i.e. access control and mappings so why be different. Mike _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 3 15:41:32 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA03657 for ; Thu, 3 Jul 2003 15:41:32 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Y9x1-00073H-Al for rddp-archive@odin.ietf.org; Thu, 03 Jul 2003 15:41:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h63Jf3Qr027103 for rddp-archive@odin.ietf.org; Thu, 3 Jul 2003 15:41:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Y9x1-000734-7V for rddp-web-archive@optimus.ietf.org; Thu, 03 Jul 2003 15:41:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA03623 for ; Thu, 3 Jul 2003 15:41:01 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Y9wz-0000N1-00 for rddp-web-archive@ietf.org; Thu, 03 Jul 2003 15:41:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Y9wz-0000My-00 for rddp-web-archive@ietf.org; Thu, 03 Jul 2003 15:41:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Y9wy-00072c-Hk; Thu, 03 Jul 2003 15:41:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Y9w2-0006vi-B2 for rddp@optimus.ietf.org; Thu, 03 Jul 2003 15:40:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA03570 for ; Thu, 3 Jul 2003 15:40:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Y9w0-0000MS-00 for rddp@ietf.org; Thu, 03 Jul 2003 15:40:00 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19Y9w0-0000M8-00 for rddp@ietf.org; Thu, 03 Jul 2003 15:40:00 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id C9E0B2458A5; Thu, 3 Jul 2003 15:39:28 -0400 (EDT) Date: Thu, 3 Jul 2003 14:39:26 -0500 From: Caitlin Bestler Subject: RE: [rddp] Security Draft issues - STag / PDs. To: Michael Krause Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <5.1.0.14.2.20030703111103.0588f540@15.13.130.73> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Thursday, July 3, 2003, krause@cup.hp.com wrote: >At 12:02 PM 7/3/2003 -0500, Caitlin Bestler wrote: >>On Wednesday, July 2, 2003, ellen.m.deleganes@intel.com wrote: >> >> >> >discussions of applicability will never be done. The full set of >> >possible local interfaces is probably infinite. However, it may be a >> >reasonable request to expand the discussions of applicability and >> >security to a larger set of "reasonable" (useful) local interfaces.> >> >>Actually it is very easy. You simply confine your discussion of >>Protection Domains to the degree that the end-to-end semantics >>require them to exist. >> >>Specifically: >> >>A Protection Domain is a container for multiple DDP Streams >>and STags. All STags within the Protection Domain are valid >>upon all DDP Streams within the Protection Domain. > >Agreed. > > >>The Protection Domain SHOULD include no more DDP Streams than >>are required by the ULP. An application SHOULD NOT include >>unrelated DDP Streams merely because there is a local desire >>to share a buffer pool. > >Conceptually true but not possible to enforce and in some >cases, the desire to share a buffer region is quite strong and >valid for the ULP in question. I do not believe these should >be included in any draft as strongly as you have shown above. > >>The local interface SHOULD NOT place pressure on the ULP >>to merge unrelated Protection Domains into a single Protection >>Domain merely to reduce RNIC resource consumption. [i.e., there >>needs to be someone to share mapping tables across multiple >>Protection Domains when referencing the same shared buffer >>region]. > >Need to separate the v-to-p mappings which may be shared with >the access controls which are the STag / PD which may or may >not be shared. Again, I don't see how what you propose is >enforced thus it would be more of a recommendation than a >SHOULD NOT. > My goal is simple. There are appliations that naturally support a large number of clients. They wish to pre-enable a large memory region (in the LMR Context / Memory Region / L-Key sense) and obtain a magic cookie from the OS that says "yes this application really has access rights to this memory, and may bind Memory Windows within it to obtain STags." These applications then want to create seperate Protection Domains for each client. And only enable each STag for that specific Protection Domain. Enabling an STag for one client may, on occassion, require enabling it for more than one DDP Stream. The application should be able to do this without being required to: a) duplicate memory registration resources, or b) expose the STag to *every* DDP Stream that is sharing the same buffer pool (or running in the same process). That is, there is no correlation between sharing local resources and sharing remote permissions. They are independent decisions. It is reasonable, and can even be secure, for an application to use one set of resources to support multiple clients. They merely need to set up their STags properly, and probably zero out buffers when first enabled for any given STag. That is much simpler, and far less restrictive, advice to given ULP developers who want to include multi-threaded servers that support a large number of clients. We need to clearly seperate the objects that control remote access (the STags and Protection Domains that are referenced in the DDP draft) from the objects that a given local interface uses to register memory and validate user access to that memory. Failure to do so can lead to confusion. I would contend that the uneccesarily restrictive advice given in the security draft is the result of exactly such confusion. And the authors of the security draft are all highly familiar with the subject matter. How much more easily would a developer using RDMA for the first time be confused? My interpretation of a SHOULD is that it is something a user can reasonably expect to be present. There may be special considerations for not providing it, which ought to be explained. You could downgrade the language to a recomendation, but I strongly feel that applications designers need to be given the assurance that they should design their interactions on the assumption that they can enable an STag on the set of DDP Streams that are part of a ULP session -- and make it no wider. It would be a disservice, if they were left with the impression that defining a Protection Domain to meet their requirements is a laudible goal it is one that they will probably not be able to achieve, and that instead they will have to choose between forcing all clients using a single server to trust each other and launching a session specific process for each client session. >>The local interface MUST allow the consumer to associate >>an STag with remote read and/or write permission for any >>contiguous subset of the consumer's address space. > >This would need to be constrained a bit. It may be for a >subset of the contiguous range - not necessarily all as one may >not be able to pin-n-map the entire range for a given operation. > What I was intending to propose is that a *minimal* local interface would at least be able to register a contiguous range of the user's virtual memory map, and that there *may* be a further ability to register non-contiguous portions or to register raw buffers that are not mapped but which the application has the appropriate permissions for anyway. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 3 17:52:31 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA08357 for ; Thu, 3 Jul 2003 17:52:31 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19YBzo-00041r-5J for rddp-archive@odin.ietf.org; Thu, 03 Jul 2003 17:52:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h63Lq4IG015482 for rddp-archive@odin.ietf.org; Thu, 3 Jul 2003 17:52:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19YBzo-00041d-0L for rddp-web-archive@optimus.ietf.org; Thu, 03 Jul 2003 17:52:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA08354 for ; Thu, 3 Jul 2003 17:52:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19YBzl-0004TV-00 for rddp-web-archive@ietf.org; Thu, 03 Jul 2003 17:52:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19YBzk-0004TS-00 for rddp-web-archive@ietf.org; Thu, 03 Jul 2003 17:52:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19YBzk-00041J-MU; Thu, 03 Jul 2003 17:52:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19YBzX-000418-OQ for rddp@optimus.ietf.org; Thu, 03 Jul 2003 17:51:47 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA08342 for ; Thu, 3 Jul 2003 17:51:44 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19YBzV-0004TJ-00 for rddp@ietf.org; Thu, 03 Jul 2003 17:51:45 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19YBzU-0004Sr-00 for rddp@ietf.org; Thu, 03 Jul 2003 17:51:44 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 2FA4324560F for ; Thu, 3 Jul 2003 17:51:16 -0400 (EDT) Date: Thu, 3 Jul 2003 16:51:12 -0500 From: Caitlin Bestler To: rddp@ietf.org X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] Protection Domain Definition Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit I believe the root of the problem with the discussion on STags and Protectin Domain can be directly traced to an incorrect definition of Protection Domain used in the proposed verbs. The protocol defines a Protection Domain as follows: > Protection Domain (PD) - A Mechanism used to associate a DDP Stream > and an STag. Under this mechanism, the use of an STag is valid > on a DDP Stream if the STag has the same Protection Domain > Identifier (PD ID) as the DDP Stream. While the verbs draft defines something else called a "Protectin Domain" > Protection Domain (PD) - A mechanism for tracking the association of > Queue Pairs, Memory Windows, and Memory Regions. PDs are > intended to be set by a Privileged Consumer to provide > protection of one process from accessing another's memory > through the use of the RNIC. The latter, I'll call them Local Protection Domains, are clearly wider than the protocols definition of a Protection Domain. This leads to two problems: a) A mistaken belief that servers that share local resources are sharing remote access rights. b) A belief that it is necessary to have a mechanism to narrow the scope of an STag to a single DDP Stream because a Protection Domain is too wide. Well, if its a Local Protection Domain, of course its too wide. If Protection Domains can be created independently of Local Protection Domains (subject only to the general rule that the Local Interface MUST NOT grant the RNIC any authority that the caller did not already have) then the ULP can already tailor an STag to be valid only for the set of DDP Stremas that are part of a single session. Secondly, if make the mistake of widening the Protection Domain to the scope implied by the Local Protection Domain then we still do not have a solution to allow the ULP to make an STag valid *only* on the DDP Streams that are part of a *single* session. A multi-threaded server could have 10,000 sessions, 30,000 DDP Streams and a *single* Local Protection Domain. If I try *real* hard, I can come up with a theoretical scenario where I can have 10,000 sessions, each with 3 DDP Streams, and there are some STags that are to be used only on a single stream. But how common is it? And if we deal with that, do we also have to deal with applications that need to allow the STag to be valid on 2 out of 3 Streams within a Protection Domain? Let's keep things simple. Allowing Protection Domain *or* QP ID carries costs: -- The bind operation must distinquish whether the STag is being bound to the entire Protection Domain or to one Stream. -- The MW query must indicate whether the STag is bound to the QP ID or to the PD ID. -- The semantics of a peer invalidate are unclear. -- There must be a constraint against a QP ID being destroyed and recreated while an STag is bound to it. Either it must be prevented, or the STag must be nullified. Either requires maintenance of a reverse link and/or claim counts. And for what benefit? If you mistake a Protection Domain for a Local Protection Domain it can appear very compelling. There will be lots of servers supporting 10,000 clients who only want an STag to be valid upon a single DDP Stream. But those can be handled by simply creating a Protection Domain for each session -- which will almost always be a single DDP Stream. Indeed many of the classic problems solved by extra connections will no longer require extra DDP Streams under RDMA. FTP data transfers are one example. My evaluation is that if Protection Domains are *properly* scoped, whether or not the local interface has its own concept of Local Protection Domain, that the number of applications that will need to control which STag is valid on which subset of the DDP STreams associated with a single session will be *very* slight -- so slight that it is not worth *any* additional complexity. The verbs draft should be updated to seperate concerns on local memory protection from ULP / protocol issues. The security draft should be updated based upon the assumption that the local interface will support Protection Domains as intended in the DDP draft. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 3 23:41:32 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA02036 for ; Thu, 3 Jul 2003 23:41:32 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19YHRY-0004zW-I1 for rddp-archive@odin.ietf.org; Thu, 03 Jul 2003 23:41:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h643f4cg019182 for rddp-archive@odin.ietf.org; Thu, 3 Jul 2003 23:41:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19YHRY-0004zJ-DI for rddp-web-archive@optimus.ietf.org; Thu, 03 Jul 2003 23:41:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA02032 for ; Thu, 3 Jul 2003 23:41:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19YHRW-0001F3-00 for rddp-web-archive@ietf.org; Thu, 03 Jul 2003 23:41:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19YHRV-0001F0-00 for rddp-web-archive@ietf.org; Thu, 03 Jul 2003 23:41:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19YHRW-0004yv-9g; Thu, 03 Jul 2003 23:41:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19YHQm-0004yX-Na for rddp@optimus.ietf.org; Thu, 03 Jul 2003 23:40:16 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA02006 for ; Thu, 3 Jul 2003 23:40:13 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19YHQk-0001EH-00 for rddp@ietf.org; Thu, 03 Jul 2003 23:40:14 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19YHQj-0001Dq-00 for rddp@ietf.org; Thu, 03 Jul 2003 23:40:13 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 7EF06244ADC for ; Thu, 3 Jul 2003 23:39:46 -0400 (EDT) Date: Thu, 3 Jul 2003 22:39:41 -0500 From: Caitlin Bestler To: rddp@ietf.org X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] verbs draft - STag validation error in 7.10.6.2 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Section 7.10.6.2 of the verbs draft has a mistake. In specifying the validation that must be done for received STags it states: > * The RNIC MUST check and enforce that the QP ID associated with > the Memory Window is equal to the QP ID associated with the > incoming remote operation that is accessing the Memory Window. If a Memory Window has "a" QP ID associated with it then it can never be valid on multiple QPs (DDP Streams). But the protocol specifically states: > Protection Domain (PD) - A Mechanism used to associate a DDP Stream > and an STag. Under this mechanism, the use of an STag is valid > on a DDP Stream if the STag has the same Protection Domain > Identifier (PD ID) as the DDP Stream. An STag with the "wrong" QP ID nevertheless has the correct PD ID. Therefore the STag is valid. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 7 10:31:33 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA27511 for ; Mon, 7 Jul 2003 10:31:33 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZX1G-0003ms-5z for rddp-archive@odin.ietf.org; Mon, 07 Jul 2003 10:31:06 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h67EV6j8014558 for rddp-archive@odin.ietf.org; Mon, 7 Jul 2003 10:31:06 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZX1G-0003mj-1v for rddp-web-archive@optimus.ietf.org; Mon, 07 Jul 2003 10:31:06 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA27483 for ; Mon, 7 Jul 2003 10:31:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ZX1D-0004oW-00 for rddp-web-archive@ietf.org; Mon, 07 Jul 2003 10:31:03 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ZX1D-0004oS-00 for rddp-web-archive@ietf.org; Mon, 07 Jul 2003 10:31:03 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZX1B-0003ll-AC; Mon, 07 Jul 2003 10:31:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZX0K-0003hv-2J for rddp@optimus.ietf.org; Mon, 07 Jul 2003 10:30:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA27413 for ; Mon, 7 Jul 2003 10:30:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ZX0H-0004o6-00 for rddp@ietf.org; Mon, 07 Jul 2003 10:30:05 -0400 Received: from fmr02.intel.com ([192.55.52.25] helo=caduceus.fm.intel.com) by ietf-mx with esmtp (Exim 4.12) id 19ZX0G-0004o3-00 for rddp@ietf.org; Mon, 07 Jul 2003 10:30:04 -0400 Received: from talaria.fm.intel.com (talaria.fm.intel.com [10.1.192.39]) by caduceus.fm.intel.com (8.11.6p2/8.11.6/d: outer.mc,v 1.66 2003/05/22 21:17:36 rfjohns1 Exp $) with ESMTP id h67ELbh07855 for ; Mon, 7 Jul 2003 14:21:38 GMT Received: from orsmsxvs041.jf.intel.com (orsmsxvs041.jf.intel.com [192.168.65.54]) by talaria.fm.intel.com (8.11.6p2/8.11.6/d: inner.mc,v 1.35 2003/05/22 21:18:01 rfjohns1 Exp $) with SMTP id h67EUMN05536 for ; Mon, 7 Jul 2003 14:30:22 GMT Received: from orsmsx331.amr.corp.intel.com ([192.168.65.56]) by orsmsxvs041.jf.intel.com (NAVGW 2.5.2.11) with SMTP id M2003070707300003459 ; Mon, 07 Jul 2003 07:30:00 -0700 Received: from orsmsx407.amr.corp.intel.com ([192.168.65.50]) by orsmsx331.amr.corp.intel.com with Microsoft SMTPSVC(5.0.2195.5329); Mon, 7 Jul 2003 07:30:00 -0700 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 Subject: RE: [rddp] verbs draft - STag validation error in 7.10.6.2 Date: Mon, 7 Jul 2003 07:29:59 -0700 Message-ID: <42050DF556283A4D977B111EB7063208074199@orsmsx407.jf.intel.com> Thread-Topic: [rddp] verbs draft - STag validation error in 7.10.6.2 Thread-Index: AcNB3iNJ7TbmN1gLSQiqbT1pGj49fgCtUnog From: "Deleganes, Ellen M" To: "Caitlin Bestler" , X-OriginalArrivalTime: 07 Jul 2003 14:30:00.0183 (UTC) FILETIME=[3F265070:01C34494] Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable Making Memory Windows only valid on a single stream in the verbs was a deliberate choice. In most programming models it makes sense to bind a window to a stream rather than to the PD. If the application writer wants a memory entity that can be remotely accessible by multiple streams, they can still use Memory Regions. This decision actually makes Memory Windows more lightweight. It is a lot quicker to do the invalidation because you only need to check incoming traffic on a single stream, instead of across all of the streams in a Protection Domain, which can be rather onerous. Ellen -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com]=20 Sent: Thursday, July 03, 2003 8:40 PM To: rddp@ietf.org Subject: [rddp] verbs draft - STag validation error in 7.10.6.2 Section 7.10.6.2 of the verbs draft has a mistake. In specifying the validation that must be done for received STags it states: > * The RNIC MUST check and enforce that the QP ID associated with=20 > the Memory Window is equal to the QP ID associated with the=20 > incoming remote operation that is accessing the Memory Window. If a Memory Window has "a" QP ID associated with it then it can never be valid on multiple QPs (DDP Streams). But the protocol specifically states: > Protection Domain (PD) - A Mechanism used to associate a DDP Stream=20 > and an STag. Under this mechanism, the use of an STag is valid=20 > on a DDP Stream if the STag has the same Protection Domain=20 > Identifier (PD ID) as the DDP Stream. An STag with the "wrong" QP ID nevertheless has the correct PD ID. Therefore the STag is valid. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 7 11:07:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA28990 for ; Mon, 7 Jul 2003 11:07:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZXa3-0005vJ-40 for rddp-archive@odin.ietf.org; Mon, 07 Jul 2003 11:07:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h67F73bM022763 for rddp-archive@odin.ietf.org; Mon, 7 Jul 2003 11:07:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZXa3-0005v1-0l for rddp-web-archive@optimus.ietf.org; Mon, 07 Jul 2003 11:07:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA28975 for ; Mon, 7 Jul 2003 11:06:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ZXa0-0005GR-00 for rddp-web-archive@ietf.org; Mon, 07 Jul 2003 11:07:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ZXZz-0005GO-00 for rddp-web-archive@ietf.org; Mon, 07 Jul 2003 11:06:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZXa1-0005tV-BH; Mon, 07 Jul 2003 11:07:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZXZM-0005sy-6I for rddp@optimus.ietf.org; Mon, 07 Jul 2003 11:06:20 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA28965 for ; Mon, 7 Jul 2003 11:06:15 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ZXZJ-0005G5-00 for rddp@ietf.org; Mon, 07 Jul 2003 11:06:17 -0400 Received: from ztxmail04.ztx.compaq.com ([161.114.1.208]) by ietf-mx with esmtp (Exim 4.12) id 19ZXZI-0005Fl-00 for rddp@ietf.org; Mon, 07 Jul 2003 11:06:16 -0400 Received: from cceexg12.americas.cpqcorp.net (cceexg12.americas.cpqcorp.net [16.110.250.124]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id 7F148AB5F for ; Mon, 7 Jul 2003 10:05:45 -0500 (CDT) Received: from cceexc16.americas.cpqcorp.net ([16.110.250.58]) by cceexg12.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Mon, 7 Jul 2003 10:05:45 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [rddp] verbs draft - STag validation error in 7.10.6.2 Date: Mon, 7 Jul 2003 10:05:44 -0500 Message-ID: <07DEBFB3DCB18A49B06B63719935867403FF5F6C@cceexc16.americas.cpqcorp.net> Thread-Topic: [rddp] verbs draft - STag validation error in 7.10.6.2 Thread-Index: AcNB3iNJ7TbmN1gLSQiqbT1pGj49fgCtUnogAACFokA= From: "Hilland, Jeff" To: X-OriginalArrivalTime: 07 Jul 2003 15:05:45.0335 (UTC) FILETIME=[3DC28870:01C34499] Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable Echoing Ellen's sentiments, this was done deliberately and with good reason. We encountered a design problem, looked at alternatives, and chose this solution due to the ease of implementation, ease of use and least perturbation to existing fabric solutions. I would prefer that we not distract the reflector traffic to Verbs when we haven't even nailed the wire protocols, but since I was the guilty party to post the Verbs draft, I'll be glad to try to answer this one and rely on our chair to determine if this conversation is premature. When dealing with a single process which has multiple clients, each represented by a different QP (stream), it is desirable to give each QP only the rights to access a small amount of memory for a small amount of time. Memory Windows are the obvious choice. But to get this to work under the old VI/IB model, this means each QP must be in it's own PD (in order to preserve security/fault domains/memory protection) and thus the Consumer would have to register memory (Register Shared, probably) for each PD - probably all of SGA. In addition, each Consumer would then have a pool of MWs for each PD. This results in a consumption of resources in the implementation in a fashion that is non-linear compared to the number of connections - and thus is wasteful. In a hardware implementation, these may (read will) be scarce resources (at least in lower end implementations). After looking at several alternatives, it was determined that the amount of time a Memory Window references a specified location is short lived. MW usage typically follows a Bind/Use/Invalidate model. So it was decided that during the Bind, the MW STag would only be valid on that one connection. Once invalidated, it would be available for general use. This allowed a model where all QPs could be in one PD, all MWs could be in one PD and you would only need to Register Memory once. This model allows for optimum use of implementation resources and matches most usage models. The Verbs draft goes through this in gory detail and suggests a method for implementing this feature. Now, if a Consumer wants to use Memory Windows across multiple connections, this flavor of Memory Windows is not optimal. This was examined as well. And by "use", I don't mean advertise, I mean use for RDMA Read/Write purposes. This means that the connections share the same access rights to advertised memory and the advertisement is long lived (otherwise Memory Protection errors would occur). And these connections have a high degree of coordination of the shared MW (again, to prevent errors that would terminate the Stream). This model begins to resemble the requirements of a fast-register or normal register model and not a MW model. So if this is a requirement, I would like to understand why none of the alternatives are acceptable (aren't memory registration models enough?) and would propose dialog on that usage model to understand the requirements. Regards, Jeff Hilland=20 -----Original Message----- From: Deleganes, Ellen M [mailto:ellen.m.deleganes@intel.com]=20 Sent: Monday, July 07, 2003 9:30 AM To: Caitlin Bestler; rddp@ietf.org Subject: RE: [rddp] verbs draft - STag validation error in 7.10.6.2 Making Memory Windows only valid on a single stream in the verbs was a deliberate choice. In most programming models it makes sense to bind a window to a stream rather than to the PD. If the application writer wants a memory entity that can be remotely accessible by multiple streams, they can still use Memory Regions. This decision actually makes Memory Windows more lightweight. It is a lot quicker to do the invalidation because you only need to check incoming traffic on a single stream, instead of across all of the streams in a Protection Domain, which can be rather onerous. Ellen -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com]=20 Sent: Thursday, July 03, 2003 8:40 PM To: rddp@ietf.org Subject: [rddp] verbs draft - STag validation error in 7.10.6.2 Section 7.10.6.2 of the verbs draft has a mistake. In specifying the validation that must be done for received STags it states: > * The RNIC MUST check and enforce that the QP ID associated with=20 > the Memory Window is equal to the QP ID associated with the=20 > incoming remote operation that is accessing the Memory Window. If a Memory Window has "a" QP ID associated with it then it can never be valid on multiple QPs (DDP Streams). But the protocol specifically states: > Protection Domain (PD) - A Mechanism used to associate a DDP Stream=20 > and an STag. Under this mechanism, the use of an STag is valid=20 > on a DDP Stream if the STag has the same Protection Domain=20 > Identifier (PD ID) as the DDP Stream. An STag with the "wrong" QP ID nevertheless has the correct PD ID. Therefore the STag is valid. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 7 12:21:31 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA01994 for ; Mon, 7 Jul 2003 12:21:31 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZYjg-00035P-IG for rddp-archive@odin.ietf.org; Mon, 07 Jul 2003 12:21:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h67GL4bG011859 for rddp-archive@odin.ietf.org; Mon, 7 Jul 2003 12:21:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZYjg-00035C-Eb for rddp-web-archive@optimus.ietf.org; Mon, 07 Jul 2003 12:21:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA01961 for ; Mon, 7 Jul 2003 12:21:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ZYjf-0006Dl-00 for rddp-web-archive@ietf.org; Mon, 07 Jul 2003 12:21:03 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ZYje-0006Di-00 for rddp-web-archive@ietf.org; Mon, 07 Jul 2003 12:21:02 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZYjd-00034X-ME; Mon, 07 Jul 2003 12:21:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZYjG-00031n-B8 for rddp@optimus.ietf.org; Mon, 07 Jul 2003 12:20:38 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA01913 for ; Mon, 7 Jul 2003 12:20:34 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ZYjE-0006DJ-00 for rddp@ietf.org; Mon, 07 Jul 2003 12:20:36 -0400 Received: from palrel12.hp.com ([156.153.255.237]) by ietf-mx with esmtp (Exim 4.12) id 19ZYjE-0006DC-00 for rddp@ietf.org; Mon, 07 Jul 2003 12:20:36 -0400 Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel12.hp.com (Postfix) with ESMTP id F36211C01614 for ; Mon, 7 Jul 2003 09:20:34 -0700 (PDT) Received: from mk731915.cup.hp.com ([15.244.171.40]) by esmail.cup.hp.com (8.11.1/8.8.6) with ESMTP id h67GFSS02664 for ; Mon, 7 Jul 2003 09:15:28 -0700 (PDT) Message-Id: <5.1.0.14.2.20030707091904.0209e520@15.13.130.73> X-Sender: krause@15.13.130.73 X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Mon, 07 Jul 2003 09:20:40 -0700 To: rddp@ietf.org From: Michael Krause Subject: Re: [rddp] verbs draft - STag validation error in 7.10.6.2 In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 10:39 PM 7/3/2003 -0500, Caitlin Bestler wrote: >Section 7.10.6.2 of the verbs draft has a mistake. In specifying the >validation that must be done for received STags it states: > > > * The RNIC MUST check and enforce that the QP ID associated with > > the Memory Window is equal to the QP ID associated with the > > incoming remote operation that is accessing the Memory Window. > >If a Memory Window has "a" QP ID associated with it then it can never >be valid on multiple QPs (DDP Streams). > >But the protocol specifically states: > > > Protection Domain (PD) - A Mechanism used to associate a DDP Stream > > and an STag. Under this mechanism, the use of an STag is valid > > on a DDP Stream if the STag has the same Protection Domain > > Identifier (PD ID) as the DDP Stream. > > >An STag with the "wrong" QP ID nevertheless has the correct PD ID. >Therefore the STag is valid. I believe Jeff's response on MW and QP association should answer this question. MW should be viewed as a narrowing of scope of a STag and does not invalidate the statement in the protocol since a MW is part of a PD. Mike _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 7 12:46:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02945 for ; Mon, 7 Jul 2003 12:46:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZZ7r-0004LY-MJ for rddp-archive@odin.ietf.org; Mon, 07 Jul 2003 12:46:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h67Gk3s0016708 for rddp-archive@odin.ietf.org; Mon, 7 Jul 2003 12:46:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZZ7r-0004LP-IM for rddp-web-archive@optimus.ietf.org; Mon, 07 Jul 2003 12:46:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02866 for ; Mon, 7 Jul 2003 12:45:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ZZ7p-0006X4-00 for rddp-web-archive@ietf.org; Mon, 07 Jul 2003 12:46:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ZZ7p-0006X1-00 for rddp-web-archive@ietf.org; Mon, 07 Jul 2003 12:46:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZZ7p-0004Kp-Jp; Mon, 07 Jul 2003 12:46:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZZ7f-0004K7-Hs for rddp@optimus.ietf.org; Mon, 07 Jul 2003 12:45:51 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02846 for ; Mon, 7 Jul 2003 12:45:47 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ZZ7d-0006WX-00 for rddp@ietf.org; Mon, 07 Jul 2003 12:45:49 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19ZZ7c-0006VG-00 for rddp@ietf.org; Mon, 07 Jul 2003 12:45:49 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 5E4A9244EBB; Mon, 7 Jul 2003 12:45:16 -0400 (EDT) Date: Mon, 7 Jul 2003 11:45:12 -0500 From: Caitlin Bestler Subject: RE: [rddp] verbs draft - STag validation error in 7.10.6.2 To: "Deleganes, Ellen M" Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <42050DF556283A4D977B111EB7063208074199@orsmsx407.jf.intel.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Monday, July 7, 2003, ellen.m.deleganes@intel.com wrote: >Making Memory Windows only valid on a single stream in the verbs was a >deliberate choice. But it was not the deliberate choice of the WG. This issue was explicitly discussed. The WG rejected both limiting STags to a single stream and allowing "any to any" matching of STags and streams. Instead, ULPs that wanted to have an STag be valid across multiple streams had to place those streams in a single Protection Domain so that the stack could perform a simple comparision rather than being required to do a lookup. >In most programming models it makes sense to bind a >window to a stream rather than to the PD. If the application writer >wants a memory entity that can be remotely accessible by multiple >streams, they can still use Memory Regions. With the verbs as proposed, they can only do so by losing the ability to dynamically define the boundaries of that region. Basically, you leave a very wide region open to nearly every stream that the server uses, or you gain byte-boundary control over a single stream. There is a very wide gap between those two. >This decision actually makes Memory Windows more lightweight. Only if you confuse Protection Domains (as defined in the protocol) with the local memory protectio mechanisms. I would submit that all of the following are "lightweight" -- A Consumer creating a "Protection Domain" identifier. -- A Consumer assigning that identifier to Memory Windows -- A Consumer assigning that identifier to a DDP Stream. -- An incoming STag is validated by comparing PD IDs. Such an interface is fully compatible with the DDP draft, easily implemented, has fewer security implications and provides the ULP with complete flexibility in creating multi-stream Sessions. >It is a lot quicker to do the invalidation because you only >need to check incoming traffic on a single stream, instead of >across all of the streams in a Protection Domain, which can be >rather onerous. > Which was specifically discussed on the reflector when Protection Domains were proposed and accepted. Specifically, that an invalidate would invalite the STag for the entire Protection Domain. Keep in mind that multi-stream STags are created *by the ULP* to server *ULP* purposes. So just as with single-stream STags, we can presume that if the ULP says to invalidate an STag that it is doing so in compliance with ULP rules. In other words, the ULP should not create a multi-stream session unless it has reason to trust all participants. Even with those restrictions, I can still see many shared-data applications where communal STags within a cluster could be of considerable value. This is particularly true for MPI apps. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 7 13:17:28 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04233 for ; Mon, 7 Jul 2003 13:17:28 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZZbq-0006au-HC for rddp-archive@odin.ietf.org; Mon, 07 Jul 2003 13:17:02 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h67HH2aM025342 for rddp-archive@odin.ietf.org; Mon, 7 Jul 2003 13:17:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZZbq-0006af-EK for rddp-web-archive@optimus.ietf.org; Mon, 07 Jul 2003 13:17:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04223 for ; Mon, 7 Jul 2003 13:16:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ZZbo-00070A-00 for rddp-web-archive@ietf.org; Mon, 07 Jul 2003 13:17:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ZZbo-000707-00 for rddp-web-archive@ietf.org; Mon, 07 Jul 2003 13:17:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZZbp-0006aL-4o; Mon, 07 Jul 2003 13:17:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZZb0-0006Ty-LN for rddp@optimus.ietf.org; Mon, 07 Jul 2003 13:16:10 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04190 for ; Mon, 7 Jul 2003 13:16:06 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ZZay-0006zE-00 for rddp@ietf.org; Mon, 07 Jul 2003 13:16:08 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19ZZay-0006yc-00 for rddp@ietf.org; Mon, 07 Jul 2003 13:16:08 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 617F924545B; Mon, 7 Jul 2003 13:15:39 -0400 (EDT) Date: Mon, 7 Jul 2003 12:15:35 -0500 From: Caitlin Bestler Subject: Re: [rddp] verbs draft - STag validation error in 7.10.6.2 To: Michael Krause , rddp@ietf.org X-Priority: 3 In-Reply-To: <5.1.0.14.2.20030707091904.0209e520@15.13.130.73> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Monday, July 7, 2003, krause@cup.hp.com wrote: >At 10:39 PM 7/3/2003 -0500, Caitlin Bestler wrote: >>Section 7.10.6.2 of the verbs draft has a mistake. In specifying the >>validation that must be done for received STags it states: >> >> > * The RNIC MUST check and enforce that the QP ID >> > associated with the Memory Window is equal to the QP ID >> > associated with the incoming remote operation that is >> > accessing the Memory Window. >> >>If a Memory Window has "a" QP ID associated with it then it can never >>be valid on multiple QPs (DDP Streams). >> >>But the protocol specifically states: >> >>> Protection Domain (PD) - A Mechanism used to associate a DDP >>> Stream and an STag. Under this mechanism, the use of an STag >>> is valid on a DDP Stream if the STag has the same Protection >>> Domain Identifier (PD ID) as the DDP Stream. >> >> >>An STag with the "wrong" QP ID nevertheless has the correct PD ID. >>Therefore the STag is valid. > >I believe Jeff's response on MW and QP association should >answer this question. MW should be viewed as a narrowing of >scope of a STag and does not invalidate the statement in the >protocol since a MW is part of a PD. > >Mike Viewing the MW as a "supplemental" feature would only work if the MR was adequate. Any review of the draft will clearly indicate that it is not. More importantly, it violates what was always an implicit capability - that the ULP can dynamically bind an STag to have precisely the boundaries that the ULP desires. A local interface that does not provide this capability, or provides it in a way that is effectively not usable, would be creating a major security concern. The key issue remains: exactly what is the "Protection Domain" referenced in the DDP draft? I maintain that it should *only* be a ULP created holder for STags and Endponts. Or more precisely, if it should hold anything else, then that should not interfere with its externally defined mission to hold STags and Endpoints. With that definition, a Protection Domain is a) lightweight. b) easy for consumers to use. c) easy for hardware to implement. d) imposes no barriers to defining the exact set of STags that are valid on a specific set of Streams, and what the boundaries and permissions for each of those STags are. e) creates no new security concerns It was specifially the fact that the security draft made incorrect statements about security vunerabilities of "DDP" that were really only security vunerabilities of *these* verbs that brought this all into focus. There are vague assertions being made that the restrictions on the application model are somehow necessary. I do not see them. The above interpretation is *totally* consistent with the DDP draft and with the discussion of Protection Domains on the reflector. Neither the verbs draft nor the assumptions made in the security draft can make that claim. If the proponents of those drafts feel that the ddp draft needs to refine its defintion of a Protection Domain then they should explicitly make the case for doing so on the reflector. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 8 09:59:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA21804 for ; Tue, 8 Jul 2003 09:59:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Zszn-0003pE-LO for rddp-archive@odin.ietf.org; Tue, 08 Jul 2003 09:59:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h68Dx39E014704 for rddp-archive@odin.ietf.org; Tue, 8 Jul 2003 09:59:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Zszn-0003p5-Hj for rddp-web-archive@optimus.ietf.org; Tue, 08 Jul 2003 09:59:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA21785 for ; Tue, 8 Jul 2003 09:58:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Zszl-0001l5-00 for rddp-web-archive@ietf.org; Tue, 08 Jul 2003 09:59:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19Zszk-0001l2-00 for rddp-web-archive@ietf.org; Tue, 08 Jul 2003 09:59:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Zszk-0003oY-Un; Tue, 08 Jul 2003 09:59:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Zsyx-0003nM-Cc for rddp@optimus.ietf.org; Tue, 08 Jul 2003 09:58:11 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA21760 for ; Tue, 8 Jul 2003 09:58:07 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Zsyv-0001kh-00 for rddp@ietf.org; Tue, 08 Jul 2003 09:58:09 -0400 Received: from palrel11.hp.com ([156.153.255.246]) by ietf-mx with esmtp (Exim 4.12) id 19Zsyu-0001kd-00 for rddp@ietf.org; Tue, 08 Jul 2003 09:58:08 -0400 Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel11.hp.com (Postfix) with ESMTP id D007C1C01806 for ; Tue, 8 Jul 2003 06:58:07 -0700 (PDT) Received: from mk731915.cup.hp.com ([15.244.168.251]) by esmail.cup.hp.com (8.11.1/8.8.6) with ESMTP id h68DqxS16244 for ; Tue, 8 Jul 2003 06:52:59 -0700 (PDT) Message-Id: <5.1.0.14.2.20030708065244.020623d0@15.13.130.73> X-Sender: krause@15.13.130.73 X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Tue, 08 Jul 2003 06:58:13 -0700 To: rddp@ietf.org From: Michael Krause Subject: Re: [rddp] verbs draft - STag validation error in 7.10.6.2 In-Reply-To: References: <5.1.0.14.2.20030707091904.0209e520@15.13.130.73> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 12:15 PM 7/7/2003 -0500, Caitlin Bestler wrote: >On Monday, July 7, 2003, krause@cup.hp.com wrote: > > >At 10:39 PM 7/3/2003 -0500, Caitlin Bestler wrote: > >>Section 7.10.6.2 of the verbs draft has a mistake. In specifying the > >>validation that must be done for received STags it states: > >> > >> > * The RNIC MUST check and enforce that the QP ID > >> > associated with the Memory Window is equal to the QP ID > >> > associated with the incoming remote operation that is > >> > accessing the Memory Window. > >> > >>If a Memory Window has "a" QP ID associated with it then it can never > >>be valid on multiple QPs (DDP Streams). > >> > >>But the protocol specifically states: > >> > >>> Protection Domain (PD) - A Mechanism used to associate a DDP > >>> Stream and an STag. Under this mechanism, the use of an STag > >>> is valid on a DDP Stream if the STag has the same Protection > >>> Domain Identifier (PD ID) as the DDP Stream. > >> > >> > >>An STag with the "wrong" QP ID nevertheless has the correct PD ID. > >>Therefore the STag is valid. > > > >I believe Jeff's response on MW and QP association should > >answer this question. MW should be viewed as a narrowing of > >scope of a STag and does not invalidate the statement in the > >protocol since a MW is part of a PD. > > > >Mike > >Viewing the MW as a "supplemental" feature would only work >if the MR was adequate. Any review of the draft will clearly >indicate that it is not. I guess we are in disagreement then. >More importantly, it violates what was always an implicit >capability - that the ULP can dynamically bind an STag >to have precisely the boundaries that the ULP desires. A MW does allow the ULP to bind the STag to precise boundaries. >A local interface that does not provide this capability, >or provides it in a way that is effectively not usable, >would be creating a major security concern. The verbs allows this to occur so I'm not sure where you are finding fault. >The key issue remains: exactly what is the "Protection >Domain" referenced in the DDP draft? > >I maintain that it should *only* be a ULP created holder >for STags and Endponts. Or more precisely, if it should >hold anything else, then that should not interfere with >its externally defined mission to hold STags and Endpoints. I don't believe anyone would disagree in this high-level view. >With that definition, a Protection Domain is > >a) lightweight. >b) easy for consumers to use. >c) easy for hardware to implement. >d) imposes no barriers to defining the exact set of STags > that are valid on a specific set of Streams, and what > the boundaries and permissions for each of those STags are. >e) creates no new security concerns > >It was specifially the fact that the security draft made >incorrect statements about security vunerabilities of >"DDP" that were really only security vunerabilities of >*these* verbs that brought this all into focus. > >There are vague assertions being made that the restrictions >on the application model are somehow necessary. I do not >see them. The above interpretation is *totally* consistent >with the DDP draft and with the discussion of Protection Domains >on the reflector. > >Neither the verbs draft nor the assumptions made in the security >draft can make that claim. If the proponents of those drafts >feel that the ddp draft needs to refine its defintion of >a Protection Domain then they should explicitly make the >case for doing so on the reflector. The only assertion made that is a point of potential disagreement is what does one validate against when a MW is being accessed. The verbs state this should be against the QP and not the PD per the logic that Jeff presented. Where is the problem? The list for PDs applies equally to MW and thus this validation process was chosen as it meets the same list of requirements. Mike _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 8 10:45:32 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA24975 for ; Tue, 8 Jul 2003 10:45:32 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZtiK-0006HJ-24 for rddp-archive@odin.ietf.org; Tue, 08 Jul 2003 10:45:06 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h68Ej4Am024128 for rddp-archive@odin.ietf.org; Tue, 8 Jul 2003 10:45:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZtiJ-0006H5-UV for rddp-web-archive@optimus.ietf.org; Tue, 08 Jul 2003 10:45:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA24959 for ; Tue, 8 Jul 2003 10:44:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ZtiH-0002FO-00 for rddp-web-archive@ietf.org; Tue, 08 Jul 2003 10:45:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ZtiG-0002FK-00 for rddp-web-archive@ietf.org; Tue, 08 Jul 2003 10:45:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ZtiH-0006GY-0m; Tue, 08 Jul 2003 10:45:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19Zti4-0006G5-Uj for rddp@optimus.ietf.org; Tue, 08 Jul 2003 10:44:48 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA24944 for ; Tue, 8 Jul 2003 10:44:44 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19Zti2-0002F5-00 for rddp@ietf.org; Tue, 08 Jul 2003 10:44:46 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19Zti1-0002Ed-00 for rddp@ietf.org; Tue, 08 Jul 2003 10:44:45 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 92C31245587; Tue, 8 Jul 2003 10:44:14 -0400 (EDT) Date: Tue, 8 Jul 2003 09:44:08 -0500 From: Caitlin Bestler Subject: Re: [rddp] verbs draft - STag validation error in 7.10.6.2 To: Michael Krause Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <5.1.0.14.2.20030708065244.020623d0@15.13.130.73> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Tuesday, July 8, 2003, krause@cup.hp.com wrote: >The only assertion made that is a point of potential >disagreement is what does one validate against when a MW is >being accessed. The verbs state this should be against the QP >and not the PD per the logic that Jeff presented. Where is the >problem? The list for PDs applies equally to MW and thus this >validation process was chosen as it meets the same list of >requirements. The verbs draft creates a lightweight mechanism to dynamically bind an STag that only works for single-stream sessions. The act of binding an MW using a QP, narrows the scope of the QP to less than the Protection Domain. This has serious implications: -- It complicates the model. In fact, for all the claims that this was a deliberate decision, I'll point out that it is not reflected in *much* of the verbs draft. For example a queery on a MW does *not* return any QP ID. This is a strange omission for a major piece of data on a well reasoned "feature". -- It creates an incentive for multi-stream sessions to NOT engage in dynamic binding and tailoring of STags. The implicit assumption seems to be that all multi-stream session ULPs are exposing their entire memory willy-nilly and no longer feel any need to do dynamic binding. -- The verbs mis-define the PD in a way that is likely to confuse application developers and minimize its proper usage. This is reflected in the poor advice given in the security draft. Specifically, the Protection Domain as defined in the DDP draft is something that serves the ULP needs. When viewing that definintion there is no doubt that the best advice is simply "make no Protection Domain wider than the ULP requires". The verbs draft entangles the DDP's Protection Domain with OS concepts of memory access authority. Those concepts probably should exist, but they are different objects. The net effect is to imply that creation of multiple Protection Domains that access the same memory is an expensive, or even wasteful, process. This is a false tradeoff. External validation does not have to be driven by the internal memory map data structures. If you merely seperate the two "Protection Domains" into the remote/protocol specified one, and the local memory-map/OS one, you get a very simple solution that imposes no artificial constraints or performance penalties on multi-stream applications. Specifically: The Local PD ID is checked to validate that an MW bind is legal, i.e. that the OS/whatever has pre-validated the poster's authority to reference memory in this mode in the target MR / memory map. The Remote PD ID is checked just as specified in the DDP draft. A local interface could even make it dynamic, so that an MW automatically assumed the Remote PD ID of the QP it was bound with, allowing MWs (as a resource) to float between sessions. I am not necessarily proposing that as a standard. Personally I think a "local interface requirements" draft is required first. But I believe that it is sufficient to show that a local interface *can* be designed that allows the full intent of the Protection Domains in the DDP draft to be supported without forcing the ULP designer to compromise security. Specifically, the local interface for binding STags -- MUST allow the tag to be bound to any contiguous subset of an eligible memory map (i.e, a registered memory region that the caller has legal access to, or that interface's equivalent). -- SHOULD be lightweight enough to encourage single-use STags. -- MUST be valid on the set of DDP Streams that the user has assigned the same Protection Domain too. A "Protection Domain" object that is independent of Page Translation Tables clearly meets these objectives. When the Protection Domain is tied directly to the OS/privilege granting authority, you end up with a choice between single-stream lightwight binds and heavyweight MR creation in order to keep an STag valid on multiple streams. To turn it around the other way, why are you defending the QP check? Why is it acceptable to block multi-stream applications from doing a lightweight bind? It is a restriction that can be shown to be unnecessary. At the minimum, security advice based upon a restriction in *one* local interface that is not inherent to the protocol has no place in the security draft. If we agree that this is a valid development strategy, then it would have to be prefaced as such. For example: "some DDP stacks may be optimized on the assumption that the Protection Domain has the same scope as a user process. On such stacks, there may be tradeoffs in supporint multiple multi-stream sessions from a single user process." But for that language to be included, a case would have to be made that optimizing the PD to match the user process made sense. I have not seen such an analysis, only an *assumption* that they were linked. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 9 02:06:33 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA29796 for ; Wed, 9 Jul 2003 02:06:33 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19a85d-0001S4-Is for rddp-archive@odin.ietf.org; Wed, 09 Jul 2003 02:06:05 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h696658f005574 for rddp-archive@odin.ietf.org; Wed, 9 Jul 2003 02:06:05 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19a85c-0001Rp-JU for rddp-web-archive@optimus.ietf.org; Wed, 09 Jul 2003 02:06:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA29270 for ; Wed, 9 Jul 2003 02:06:01 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19a85Z-0003q4-00 for rddp-web-archive@ietf.org; Wed, 09 Jul 2003 02:06:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19a85Y-0003q1-00 for rddp-web-archive@ietf.org; Wed, 09 Jul 2003 02:06:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19a85Z-0001RN-RQ; Wed, 09 Jul 2003 02:06:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19a84x-0001JZ-Uu for rddp@optimus.ietf.org; Wed, 09 Jul 2003 02:05:23 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id CAA28697 for ; Wed, 9 Jul 2003 02:05:21 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19a84u-0003pn-00 for rddp@ietf.org; Wed, 09 Jul 2003 02:05:20 -0400 Received: from mxic2.corp.emc.com ([128.221.31.40]) by ietf-mx with esmtp (Exim 4.12) id 19a84t-0003pj-00 for rddp@ietf.org; Wed, 09 Jul 2003 02:05:19 -0400 Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Wed, 9 Jul 2003 02:04:49 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564CEBC@corpmx14.us.dg.com> To: rddp@ietf.org Date: Wed, 9 Jul 2003 01:55:15 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] RDDP preliminary agenda Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a bit later than I would have liked, due to the US holiday weekend. Please get comments to me quickly, as I need to get this to the secretariat ASAP. Also - would whomever is going to plan to talk about the security draft please get in touch with me, as there's a fair amount of time allocated to it to take up security issues. Thanks, --David Vienna IETF Meeting July 13-18, 2003 ------------------- Remote Direct Data Placement WG (rddp) CHAIR: David L. Black The RDDP WG works on Remote Direct Data Placement and Remote Direct Memory Access functionality. For more information, please refer to the rddp charter at: http://www.ietf.org/html.charters/rddp-charter.html AGENDA (subject to change): Tuesday, July 15, 1700-1800 ---------------------------- - Agenda Bashing and Administrivia (5 min) - Blue Sheets - NOTE WELL - Status of drafts (5 min) - SCTP Mapping (5 min) (draft-stewart-rddp-sctp-02.txt) - Applicability Statement (5 min) (draft-ietf-rddp-applicability-00.txt) - Last Call Issue: Shared Receive Queues vs. Shared Buffer Pools (20 min) (draft-ietf-rddp-arch-02.txt) (draft-ietf-rddp-problem-statement-02.txt) This issue that arose on the list around the architecture and problem statement drafts. This is the only issue from WG Last Call - once resolved, the drafts will be submitted to the ADs/IESG. - Security (20 min) (draft-rddp-pinkerton-security-00.txt) Overview of approach and issues. A request will be made to accept a revised version of this draft as an official WG draft. RDDP WG Thursday, July 17, 1530-1730 ----------------------------- - Agenda Re-Bashing and Administrivia (5 min) - Security (continued) (30 min) - TCP mapping rough consensus status review (20 min) WG chair review and discussion of resolution of major issues in the TCP mapping. - TCP mapping (65 min) (draft-culley-iwarp-mpa-03.txt) Based on progress achieved on the mailing list, a request will be made to accept a revised version of this draft as an official WG draft. NOTE: Attendees should also read the following draft as background for the meeting, although no agenda time is allocated to it: (1) draft-ietf-rddp-rdma-concerns-00.txt Describes concerns that rddp work needs to address. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 9 19:32:32 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA29034 for ; Wed, 9 Jul 2003 19:32:32 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aOPt-0001KS-D8 for rddp-archive@odin.ietf.org; Wed, 09 Jul 2003 19:32:05 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h69NW55g005093 for rddp-archive@odin.ietf.org; Wed, 9 Jul 2003 19:32:05 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aOPt-0001K4-6I for rddp-web-archive@optimus.ietf.org; Wed, 09 Jul 2003 19:32:05 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA29020 for ; Wed, 9 Jul 2003 19:32:01 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19aOPr-0005FW-00 for rddp-web-archive@ietf.org; Wed, 09 Jul 2003 19:32:03 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19aOPr-0005FS-00 for rddp-web-archive@ietf.org; Wed, 09 Jul 2003 19:32:03 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aOPq-0001JU-O7; Wed, 09 Jul 2003 19:32:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aOPp-0001J1-AP for rddp@optimus.ietf.org; Wed, 09 Jul 2003 19:32:01 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA29003 for ; Wed, 9 Jul 2003 19:31:57 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19aOPn-0005FA-00 for rddp@ietf.org; Wed, 09 Jul 2003 19:31:59 -0400 Received: from maho3msx2.isus.emc.com ([128.221.11.32] helo=maho3msx2.corp.emc.com) by ietf-mx with esmtp (Exim 4.12) id 19aOPm-0005EG-00 for rddp@ietf.org; Wed, 09 Jul 2003 19:31:58 -0400 Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <3G4LBLCX>; Wed, 9 Jul 2003 19:31:22 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564CEC4@corpmx14.us.dg.com> To: rddp@ietf.org Subject: RE: [rddp] RDDP preliminary agenda Date: Wed, 9 Jul 2003 19:21:46 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , The preliminary agenda is now ~final and has been sent to the IETF secretariat for posting on the web site. If there are any problems/concerns with the agenda, we can still bash it in Vienna. Thanks, --David > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Wednesday, July 09, 2003 1:55 AM > To: rddp@ietf.org > Subject: [rddp] RDDP preliminary agenda > > > This is a bit later than I would have liked, due to the > US holiday weekend. Please get comments to me quickly, > as I need to get this to the secretariat ASAP. > > Also - would whomever is going to plan to talk about > the security draft please get in touch with me, as > there's a fair amount of time allocated to it to take > up security issues. > > Thanks, > --David > > Vienna IETF Meeting > July 13-18, 2003 > ------------------- > > Remote Direct Data Placement WG (rddp) > > CHAIR: David L. Black > > The RDDP WG works on Remote Direct Data Placement and Remote > Direct Memory Access functionality. For more information, > please refer to the rddp charter at: > > http://www.ietf.org/html.charters/rddp-charter.html > > AGENDA (subject to change): > > Tuesday, July 15, 1700-1800 > ---------------------------- > - Agenda Bashing and Administrivia (5 min) > - Blue Sheets > - NOTE WELL > > - Status of drafts (5 min) > > - SCTP Mapping (5 min) > (draft-stewart-rddp-sctp-02.txt) > > - Applicability Statement (5 min) > (draft-ietf-rddp-applicability-00.txt) > > - Last Call Issue: Shared Receive Queues vs. Shared Buffer > Pools (20 min) > (draft-ietf-rddp-arch-02.txt) > (draft-ietf-rddp-problem-statement-02.txt) > > This issue that arose on the list around the > architecture and problem statement drafts. This is > the only issue from WG Last Call - once resolved, the > drafts will be submitted to the ADs/IESG. > > - Security (20 min) > (draft-rddp-pinkerton-security-00.txt) > > Overview of approach and issues. A request will be made > to accept a revised version of this draft as an official > WG draft. > > > RDDP WG Thursday, July 17, 1530-1730 > ----------------------------- > - Agenda Re-Bashing and Administrivia (5 min) > > - Security (continued) (30 min) > > - TCP mapping rough consensus status review (20 min) > WG chair review and discussion of resolution of > major issues in the TCP mapping. > > - TCP mapping (65 min) > (draft-culley-iwarp-mpa-03.txt) > > Based on progress achieved on the mailing list, a request > will be made to accept a revised version of this draft as > an official WG draft. > > > NOTE: Attendees should also read the following draft as background > for the meeting, although no agenda time is allocated to it: > > (1) draft-ietf-rddp-rdma-concerns-00.txt > Describes concerns that rddp work needs to address. > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 9 20:00:31 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA29832 for ; Wed, 9 Jul 2003 20:00:31 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aOqx-0003N3-IH for rddp-archive@odin.ietf.org; Wed, 09 Jul 2003 20:00:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6A003V1012949 for rddp-archive@odin.ietf.org; Wed, 9 Jul 2003 20:00:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aOqx-0003Mm-CN for rddp-web-archive@optimus.ietf.org; Wed, 09 Jul 2003 20:00:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA29811 for ; Wed, 9 Jul 2003 20:00:01 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19aOqv-0005VL-00 for rddp-web-archive@ietf.org; Wed, 09 Jul 2003 20:00:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19aOqv-0005VI-00 for rddp-web-archive@ietf.org; Wed, 09 Jul 2003 20:00:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aOqv-0003ME-Ir; Wed, 09 Jul 2003 20:00:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aOqn-0003LU-Nn for rddp@optimus.ietf.org; Wed, 09 Jul 2003 19:59:53 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA29808 for ; Wed, 9 Jul 2003 19:59:51 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19aOql-0005VB-00 for rddp@ietf.org; Wed, 09 Jul 2003 19:59:51 -0400 Received: from [168.159.2.8] (helo=mailhub.lss.emc.com) by ietf-mx with esmtp (Exim 4.12) id 19aOql-0005V8-00 for rddp@ietf.org; Wed, 09 Jul 2003 19:59:51 -0400 Received: from mxic1.corp.emc.com (mxic1.isus.emc.com [168.159.129.100]) by mailhub.lss.emc.com (Switch-2.2.5/Switch-2.2.0) with ESMTP id h69NxmR24374 for ; Wed, 9 Jul 2003 19:59:48 -0400 (EDT) Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Wed, 9 Jul 2003 19:59:47 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564CEC5@corpmx14.us.dg.com> To: rddp@ietf.org Date: Wed, 9 Jul 2003 19:50:11 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] RDDP Verbs Draft Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , FYI, Since this draft (draft-hilland-rddp-verbs-00.txt) has come up on the list, I want to explain a few things about the draft's status and appropriate role/usage in this WG. I don't think anything that has been posted to the list is significantly at odds with what I have to say, but I thought some guidance from the WG chair might be appreciated. (1) The verbs draft is not an official RDDP WG draft, and is not likely to become one soon. The reasons for this include some of the rest of this message and the facts that both the ADs and WG chair expect the RDDP WG to finish much of the work currently on its plate before undertaking significant new work. The verbs draft will take some effort to deal with, as it's not only enormous (200+ pages), but also breaks some new ground for the IETF in specifying what's essentially an API template. (2) It's reasonable for RDDP WG members to look at the verbs draft to understand how one group of implementers intends to implement RDDP. It is unreasonable to assume that the verbs in the draft will be the only way to implement RDDP or the only acceptable/ allowed local interface to RDDP functionality. (3) As a consequence, the verbs draft cannot be the repository of any required functional behavior for RDDP (e.g., that is essential to the security properties of RDDP protocols or ensuring that RDDP ULPs have adequate security means available to protect themselves). The RDDP protocol drafts and the RDDP security draft MUST stand on their own, and will not be permitted to make normative references to the verbs draft. (4) It is reasonable to specify local functionality that is required for correct usage, integrity, security, etc. of the RDDP protocols. In addition to the "TCP delivers to the ULP in order" requirement that we're very familiar with, RFC 2401 contains some significant functional requirements on local IPsec implementations that are necessary to ensure the security properties of the IPsec protocols (see the material on the Security Policy Database and Security Association Database in Sections 4 and 5). To the extent that this is necessary for RDDP, I would encourage brevity of specification (i.e., length should be much closer to the 20+ pages devoted to this in RFC 2401 rather than the 200+ pages in the verbs draft). Comments and questions on this can be sent to the list or to me directly - I'll do my best to respond to all of them. Thanks, --David (RDDP WG Chair) ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 9 20:09:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA00159 for ; Wed, 9 Jul 2003 20:09:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aOze-0004Rh-GB for rddp-archive@odin.ietf.org; Wed, 09 Jul 2003 20:09:02 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6A0926e017083 for rddp-archive@odin.ietf.org; Wed, 9 Jul 2003 20:09:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aOze-0004RS-DF for rddp-web-archive@optimus.ietf.org; Wed, 09 Jul 2003 20:09:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA00137 for ; Wed, 9 Jul 2003 20:09:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19aOzc-0005Zy-00 for rddp-web-archive@ietf.org; Wed, 09 Jul 2003 20:09:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19aOzb-0005Zv-00 for rddp-web-archive@ietf.org; Wed, 09 Jul 2003 20:08:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aOzc-0004R7-Pp; Wed, 09 Jul 2003 20:09:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aOzP-0004Qi-J1 for rddp@optimus.ietf.org; Wed, 09 Jul 2003 20:08:47 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA00112 for ; Wed, 9 Jul 2003 20:08:45 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19aOzN-0005Zj-00 for rddp@ietf.org; Wed, 09 Jul 2003 20:08:45 -0400 Received: from mxic2.corp.emc.com ([128.221.31.40]) by ietf-mx with esmtp (Exim 4.12) id 19aOzM-0005ZX-00 for rddp@ietf.org; Wed, 09 Jul 2003 20:08:45 -0400 Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Wed, 9 Jul 2003 20:08:15 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564CEC6@corpmx14.us.dg.com> To: rddp@ietf.org Date: Wed, 9 Jul 2003 19:58:39 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] CRC Rough Consensus recap Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , I thought I sent this out to the list a week or so ago, but some helpful mailer seems to have found it entirely too tasty ... I believe the following items represent the rough consensus of the RDDP WG on CRC requirements for the TCP mapping of the RDDP protocols: (A) CRC support MUST be implemented. (B) CRC usage is all-or-nothing on header and data. There is no requirement for CRC on header but not on data or CRC on data but not on header. NB: This does not imply that there is a single CRC vs. split header and data CRCs, as there are some additional issues bearing on this. (C) CRC usage is bidirectional. Either CRCs are used in both directions on a connection, or in neither. There is no requirement for CRC usage in one direction and not the other. (D) CRC omission requires consent of both ends of the connection. If either end of the connection specifies that CRCs are to be used, then they MUST be used [in both directions (C)]. This entails some sort of mechanism to enforce CRC usage when the two ends of the connection start out with different preferences (or break the connection if the different preference have created a situation from which it is impossible to continue). (E) CRCs are OPTIONAL to use, but (per (D) above), either end of the connection can force CRC usage in both directions. (E') CRC usage is an administrative configuration decision - it is not intended to be visible to ULPs. This represents an affirmation of the consensus calls on (B), (E) and (E'), as I have seen no objections to them since they were posted. As noted earlier, there has been considerable elaboration on (E') - the right place to capture that elaboration and tell ULP implementers what they should expect and assume is the applicability statement (draft-ietf-rddp-applicability-00.txt). Please keep this in mind in reviewing that draft for discussion in Vienna. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 9 20:44:31 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01102 for ; Wed, 9 Jul 2003 20:44:31 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aPXX-0006Qx-PU for rddp-archive@odin.ietf.org; Wed, 09 Jul 2003 20:44:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6A0i3EJ024725 for rddp-archive@odin.ietf.org; Wed, 9 Jul 2003 20:44:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aPXX-0006Qi-KX for rddp-web-archive@optimus.ietf.org; Wed, 09 Jul 2003 20:44:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01099 for ; Wed, 9 Jul 2003 20:44:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19aPXV-0005o5-00 for rddp-web-archive@ietf.org; Wed, 09 Jul 2003 20:44:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19aPXU-0005o2-00 for rddp-web-archive@ietf.org; Wed, 09 Jul 2003 20:44:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aPXU-0006QO-La; Wed, 09 Jul 2003 20:44:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aPX3-0006H4-57 for rddp@optimus.ietf.org; Wed, 09 Jul 2003 20:43:33 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01094 for ; Wed, 9 Jul 2003 20:43:30 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19aPX0-0005np-00 for rddp@ietf.org; Wed, 09 Jul 2003 20:43:30 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19aPX0-0005nT-00 for rddp@ietf.org; Wed, 09 Jul 2003 20:43:30 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 9E315245623 for ; Wed, 9 Jul 2003 20:42:59 -0400 (EDT) Date: Wed, 9 Jul 2003 19:42:54 -0500 From: Caitlin Bestler To: rddp@ietf.org X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] Ordering Clarification on STag Invalidation Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On page 52 of the RDMAP draft, the following is specified about processing of STag Invalidations: > 9. An STag that is associated with an RDMAP Stream becomes > invalid upon reception of a valid Send with Invalidate or Send > with Solicited Event Message. RDMAP MUST invalidate the STag > sent in a valid Send with Invalidate or Send with Solicited > Event and Invalidate Message, before Completing the Send with > Invalidate or Send with Solicited Event and Invalidate > Message. My assumption is that the same requirements on completing the send at the Data Sink also apply to the invalidate: that all prior messages on the same stream must have been placed and completed. Which essentially means that the STag Invalidation should take place immediately before generating the completion. To say otherwise would mean that the invalidate could clobber the Tagged put that was earlier in the stream but was awaiting retransmission. So there isn't much doubt about the intent here. My question is, do we have this nailed down in some other language? I was looking and couldn't find it. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 10 22:47:31 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA01233 for ; Thu, 10 Jul 2003 22:47:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19anw7-0001k8-Mb for rddp-archive@odin.ietf.org; Thu, 10 Jul 2003 22:47:06 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6B2l3Eu006696 for rddp-archive@odin.ietf.org; Thu, 10 Jul 2003 22:47:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19anw7-0001jv-IA for rddp-web-archive@optimus.ietf.org; Thu, 10 Jul 2003 22:47:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA01229 for ; Thu, 10 Jul 2003 22:46:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19anw4-0001aG-00 for rddp-web-archive@ietf.org; Thu, 10 Jul 2003 22:47:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19anw3-0001aC-00 for rddp-web-archive@ietf.org; Thu, 10 Jul 2003 22:46:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19anw4-0001jZ-QH; Thu, 10 Jul 2003 22:47:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19anvN-0001Xv-W0 for rddp@optimus.ietf.org; Thu, 10 Jul 2003 22:46:18 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA01224 for ; Thu, 10 Jul 2003 22:46:12 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19anvK-0001a9-00 for rddp@ietf.org; Thu, 10 Jul 2003 22:46:14 -0400 Received: from atlrel7.hp.com ([156.153.255.213]) by ietf-mx with esmtp (Exim 4.12) id 19anvJ-0001a6-00 for rddp@ietf.org; Thu, 10 Jul 2003 22:46:14 -0400 Received: from rosemail.rose.hp.com (rosemail.rose.hp.com [15.96.64.26]) by atlrel7.hp.com (Postfix) with ESMTP id E51131C004BC for ; Thu, 10 Jul 2003 22:46:08 -0400 (EDT) Received: from swathi (pal1nai162246.nsr.hp.com [15.244.162.246]) by rosemail.rose.hp.com with SMTP (8.7.1/8.7.3 SMKit7.02) id TAA25980 for ; Thu, 10 Jul 2003 19:46:07 -0700 (PDT) Message-ID: <00a601c34756$9365aea0$66a0f40f@rose.hp.com> From: "Mallikarjun C." To: Date: Thu, 10 Jul 2003 19:45:19 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Content-Transfer-Encoding: 7bit Subject: [rddp] Review comment on DDP untagged buffer model Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit I have a comment on the DDP untagged buffer model of the current DDP spec. The DDP spec implies that the DDP Stream is to be dropped by implementations when there is no receive buffer to place an inbound untagged message. This is mandating a specific type of implementation, and IMHO, is needless. The DDP spec should instead allow for implementation flexibility in this area. Those RNICs that can gracefully deal with lack of receive buffers via LLP means (such as TCP retransmit) should be allowed as well. I can agree with the authors' original intent that ULP-level flow control and/or adequate buffer provisioning must be done to avoid needless retransmits, but mandating a connection drop on the first instance of hitting a (most likely a very transient) no-buffer window is severe. I believe the DDP spec can allow this RNIC implementation flexibility with minimal text changes. My review uncovered only two areas of text that mandated the connection drop requirement. section 3.2 (Protocol Overview), " Note, however, that the ULP needs to address flow control issues because if a DDP Message arrives for an Untagged Buffer without an associated receive ULP buffer, the DDP Message is dropped, the DDP Stream is disabled for reception, and an error is reported to the ULP at the Data Sink." and section 9.1 (Errors detected at the Data Sink). " 2. The QN and MSN have an associated buffer that allows Placement of the payload." My suggestion is to change the text along the following lines: Section 3.2 to: " Note, however, that the ULP needs to address flow control issues." and section 9.1 to: " 2. The QN and MSN have an associated buffer that allows Placement of the payload. DDP implementations MAY optionally consider lack of an associated buffer to be not an error, but such implementations MUST ensure that the DDP Segment is eventually recovered using LLP means in a ULP-transparent way." -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Hewlett-Packard MS 5668 Roseville CA 95747 cbm@rose.hp.com _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 10 23:28:31 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA02113 for ; Thu, 10 Jul 2003 23:28:31 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aoZp-0004UX-U0 for rddp-archive@odin.ietf.org; Thu, 10 Jul 2003 23:28:05 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6B3S5MU017253 for rddp-archive@odin.ietf.org; Thu, 10 Jul 2003 23:28:05 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aoZo-0004Ty-Bb for rddp-web-archive@optimus.ietf.org; Thu, 10 Jul 2003 23:28:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA02101 for ; Thu, 10 Jul 2003 23:27:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19aoZm-0001nK-00 for rddp-web-archive@ietf.org; Thu, 10 Jul 2003 23:28:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19aoZl-0001nH-00 for rddp-web-archive@ietf.org; Thu, 10 Jul 2003 23:28:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aoZm-0004TP-6x; Thu, 10 Jul 2003 23:28:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19aoZN-0004SM-Mx for rddp@optimus.ietf.org; Thu, 10 Jul 2003 23:27:37 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA02096 for ; Thu, 10 Jul 2003 23:27:30 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19aoZJ-0001nE-00 for rddp@ietf.org; Thu, 10 Jul 2003 23:27:33 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19aoZJ-0001n4-00 for rddp@ietf.org; Thu, 10 Jul 2003 23:27:33 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 854BD24503A; Thu, 10 Jul 2003 23:27:06 -0400 (EDT) Date: Thu, 10 Jul 2003 22:27:03 -0500 From: Caitlin Bestler Subject: Re: [rddp] Review comment on DDP untagged buffer model To: "Mallikarjun C." Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <00a601c34756$9365aea0$66a0f40f@rose.hp.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit There are two distinct issues when dealing with the Untagged Message buffer model that must be dealt with seperately: flow control and provisioning of buffers. Requiring the Data Sink to *always* prepost a buffer to enable each reception of each untagged message is clearly wasteful. I posted an objection to the last call on the architecture document on this point earlier. Requiring pre-committed buffers for all eligible untagged messages simply takes too much memory. A server could easily have 50,000 clients and allow any one of them to have 6 messages in flight. With the simple "one buffer - one authorized MSN" rule you would have to commit 300,000 buffers. But statistics and/or the aggregate bandwdith of your connection to those 50,000 clients might mean that only 50,000 buffers were needed. So the protocol needs to allow buffers to be under-provisioned, that is taken from a common pool, should the local interface choose to offer such a capability. It certainly would not be a mandatory feature, it really only applies when you have a large number of connections. But there is a second aspect - flow control. Each untagged message needs to be authorized by the Data Sink through some mechanism. Typically this is based upon a session established number of "send credits". Each untagged message from the Data Source consumes a credit. A reverse untagged message typically restores it (although other ULP mechanisms could be imagined, the NFS drafts allow the credits to be adjusted by an amount other than one in the reply, for exmaple). The question is, who enforces this limit. The traditional approach is to require the Data Sink to pre-enable each receive by posting a buffer. I believe that should be relaxed only to the extent of eliminating the need for a buffer to be supplied. The Data Sink ULP would still be required to authorize each reception, but it MAY ask that the buffer come from some specified pool or even a system pool (at the discretion of the local interface as to whether such a resource is available, and if available whether this user can use it). Requiring the ULP to authorize each untagged message in advance has several advantages: -- it allows the receiving endpoint to bound the range of valid MSNs simply. This range is a very critical resource, the EP must track the receive status of every message within the eligible window. If an errant peer jumps ahead 100 MSNs, then the EP has to track the completion status of all 100 intermediate messages. -- If a buffer pool is in fact use, it prevents a single peer from draining that pool. Keep in mind that traditional buffer based flow control does not apply to RDMA streams. This is dealt with in the applicability draft in greater detail. But effectively, ULP pacing is more critical for RDMA applications than for non-RDMA applications that have the "benefit" of being paced by traditional transport layer bottlenecks. Allowing statistical pooling of buffers is undoubtedly of benefit to many implementations -- and the protocol should clearly allow this. But I disagree that it is a "benefit" to be forgiving to remote peers that are sending more untagged messages than they are allowed. It is actually a performance and security concern. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Jul 11 03:16:37 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA19102 for ; Fri, 11 Jul 2003 03:16:37 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19as8W-0005eb-Pp for rddp-archive@odin.ietf.org; Fri, 11 Jul 2003 03:16:09 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6B7G8dk021730 for rddp-archive@odin.ietf.org; Fri, 11 Jul 2003 03:16:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19as8W-0005eN-Ih for rddp-web-archive@optimus.ietf.org; Fri, 11 Jul 2003 03:16:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA19086 for ; Fri, 11 Jul 2003 03:16:01 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19as8P-00034E-00 for rddp-web-archive@ietf.org; Fri, 11 Jul 2003 03:16:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19as8P-00034B-00 for rddp-web-archive@ietf.org; Fri, 11 Jul 2003 03:16:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19as8P-0005dl-Qo; Fri, 11 Jul 2003 03:16:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19as7T-0005ci-B8 for rddp@optimus.ietf.org; Fri, 11 Jul 2003 03:15:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA19060; Fri, 11 Jul 2003 03:15:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19as7R-000342-00; Fri, 11 Jul 2003 03:15:01 -0400 Received: from e34.co.us.ibm.com ([32.97.110.132]) by ietf-mx with esmtp (Exim 4.12) id 19as7Q-00033z-00; Fri, 11 Jul 2003 03:15:00 -0400 Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e34.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h6B7EUDG266586; Fri, 11 Jul 2003 03:14:30 -0400 Received: from d03nm115.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h6B7ETRL194302; Fri, 11 Jul 2003 01:14:29 -0600 To: rddp@ietf.org, rddp-admin@ietf.org MIME-Version: 1.0 Subject: Re: [rddp] Review comment on DDP untagged buffer model X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 From: John Hufferd Message-ID: Date: Fri, 11 Jul 2003 00:14:24 -0700 X-MIMETrack: Serialize by Router on D03NM115/03/M/IBM(Release 6.0.1 [IBM]|June 10, 2003) at 07/11/2003 01:14:29, Serialize complete at 07/11/2003 01:14:29 Content-Type: multipart/alternative; boundary="=_alternative 0027C12B88256D60_=" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multipart message in MIME format. --=_alternative 0027C12B88256D60_= Content-Type: text/plain; charset="us-ascii" I completely agree with Mallikarjun. There are straight forward techniques to permit the LLP to notice that the buffer is not yet available, and drop the buffer, thus permitting exploitation of the basic retry capability of the LLP. I am not saying that it should be required but only that it should be permitted, as long as it is transparent to the ULP. When ever there is a buffer replenishment process from a pool of buffers, there is the possibility that because of some hiccup in the system (perhaps a higher priority task in the OS being dispatch to handle some error recovery process), the replenishment process may not be able to replenish the pool in time. As a result you would have to way over compensate by adding and undefined number of buffers, just in case something uncontrollable happens. It is better to have an RNIC that can tolerate a temporary shortage. Therefore, I agree with Mallikarjun and feel we should NOT mandate that the link be dropped if the required buffer is missing. It should be clearly specified that the RNIC could act that way, but it should not be a hard requirement that the RNIC must operate in that manner. In fact if an RNIC vendor offers such a retry capability, there may be a number of System vendors that would like to use that RNICs so that the applications, which run on their systems, will not be needlessly terminated when high priority system processes are invoked. . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/System Group, San Jose CA Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Alt Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com "Mallikarjun C." Sent by: rddp-admin@ietf.org 07/10/2003 07:45 PM To: cc: Subject: [rddp] Review comment on DDP untagged buffer model I have a comment on the DDP untagged buffer model of the current DDP spec. The DDP spec implies that the DDP Stream is to be dropped by implementations when there is no receive buffer to place an inbound untagged message. This is mandating a specific type of implementation, and IMHO, is needless. The DDP spec should instead allow for implementation flexibility in this area. Those RNICs that can gracefully deal with lack of receive buffers via LLP means (such as TCP retransmit) should be allowed as well. I can agree with the authors' original intent that ULP-level flow control and/or adequate buffer provisioning must be done to avoid needless retransmits, but mandating a connection drop on the first instance of hitting a (most likely a very transient) no-buffer window is severe. I believe the DDP spec can allow this RNIC implementation flexibility with minimal text changes. My review uncovered only two areas of text that mandated the connection drop requirement. section 3.2 (Protocol Overview), " Note, however, that the ULP needs to address flow control issues because if a DDP Message arrives for an Untagged Buffer without an associated receive ULP buffer, the DDP Message is dropped, the DDP Stream is disabled for reception, and an error is reported to the ULP at the Data Sink." and section 9.1 (Errors detected at the Data Sink). " 2. The QN and MSN have an associated buffer that allows Placement of the payload." My suggestion is to change the text along the following lines: Section 3.2 to: " Note, however, that the ULP needs to address flow control issues." and section 9.1 to: " 2. The QN and MSN have an associated buffer that allows Placement of the payload. DDP implementations MAY optionally consider lack of an associated buffer to be not an error, but such implementations MUST ensure that the DDP Segment is eventually recovered using LLP means in a ULP-transparent way." -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Hewlett-Packard MS 5668 Roseville CA 95747 cbm@rose.hp.com _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp --=_alternative 0027C12B88256D60_= Content-Type: text/html; charset="us-ascii"
I completely agree with Mallikarjun.  There are straight forward techniques to permit the LLP to notice that the buffer is not yet available, and drop the buffer, thus permitting  exploitation  of the basic retry capability of the LLP.  

I am not saying that it should be required but only that it should be permitted, as long as it is transparent to the ULP.

When ever there is a buffer replenishment process from a pool of buffers, there is the possibility that because of some hiccup in the system (perhaps a higher priority task in the OS being dispatch to handle some error recovery process), the replenishment process may not be able to replenish the pool in time.  As a result you would have to way over compensate by adding and undefined number of buffers, just in case something uncontrollable happens.  It is better  to have an RNIC that can tolerate a temporary shortage.

Therefore, I agree with Mallikarjun and feel we should NOT mandate that the link be dropped if the required buffer is missing.  It should be clearly specified that the RNIC could act that way, but it should not be a hard requirement that the RNIC must operate in that manner.  In fact if an RNIC vendor offers such a retry capability, there may be a number of System vendors that would like to use that  RNICs so that the applications, which run on their systems, will not be needlessly terminated when high priority system processes are invoked.
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: hufferd@us.ibm.com


"Mallikarjun C." <cbm@rose.hp.com>
Sent by: rddp-admin@ietf.org

07/10/2003 07:45 PM

       
        To:        <rddp@ietf.org>
        cc:        
        Subject:        [rddp] Review comment on DDP untagged buffer model



I  have a comment on the DDP untagged buffer model
of the current DDP spec.

The DDP spec implies that the DDP Stream is to be
dropped by implementations when there is no receive
buffer to place an inbound untagged message.  This
is mandating a specific type of implementation, and IMHO,
is needless.  The DDP spec should instead allow for
implementation flexibility in this area.  Those RNICs
that can gracefully deal with lack of receive buffers
via LLP means (such as TCP retransmit) should be
allowed as well.  I can agree with the authors' original intent
that ULP-level flow control and/or adequate buffer provisioning
must be done to avoid needless retransmits, but mandating
a connection drop on the first instance of hitting a (most
likely a very transient) no-buffer window is severe.

I believe the DDP spec can allow this RNIC implementation
flexibility with minimal text changes.  My review uncovered
only two areas of text that mandated the connection drop
requirement.

section 3.2 (Protocol Overview),
   " Note, however, that
       the ULP needs to address flow control issues because if a DDP
       Message arrives for an Untagged Buffer without an associated
       receive ULP buffer, the DDP Message is dropped, the DDP Stream
       is disabled for reception, and an error is reported to the ULP
       at the Data Sink."

and section 9.1  (Errors detected at the Data Sink).
   "   2.  The QN and MSN have an associated buffer that allows Placement
      of the payload."

My suggestion is to change the text along the following lines:
Section 3.2 to:
   " Note, however, that the ULP needs to address flow control issues."
and section 9.1 to:
   " 2. The QN and MSN have an associated buffer that allows Placement of the payload. DDP implementations
MAY optionally consider lack of an associated buffer to be not an error, but such implementations MUST ensure
that the DDP Segment is eventually recovered using LLP means in a ULP-transparent way."
--
Mallikarjun

Mallikarjun Chadalapaka
Networked Storage Architecture
Network Storage Solutions
Hewlett-Packard MS 5668
Roseville CA 95747
cbm@rose.hp.com



_______________________________________________
rddp mailing list
rddp@ietf.org
https://www1.ietf.org/mailman/listinfo/rddp

--=_alternative 0027C12B88256D60_=-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Jul 11 08:31:31 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA27304 for ; Fri, 11 Jul 2003 08:31:31 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ax3H-0005qC-M4 for rddp-archive@odin.ietf.org; Fri, 11 Jul 2003 08:31:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6BCV3nY022448 for rddp-archive@odin.ietf.org; Fri, 11 Jul 2003 08:31:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ax3H-0005pz-AW for rddp-web-archive@optimus.ietf.org; Fri, 11 Jul 2003 08:31:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA27268 for ; Fri, 11 Jul 2003 08:31:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ax3G-0005IJ-00 for rddp-web-archive@ietf.org; Fri, 11 Jul 2003 08:31:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ax3F-0005IG-00 for rddp-web-archive@ietf.org; Fri, 11 Jul 2003 08:31:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ax3E-0005pb-Uh; Fri, 11 Jul 2003 08:31:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ax2O-0005er-TO for rddp@optimus.ietf.org; Fri, 11 Jul 2003 08:30:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA27248 for ; Fri, 11 Jul 2003 08:30:05 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ax2N-0005Hv-00 for rddp@ietf.org; Fri, 11 Jul 2003 08:30:07 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19ax2N-0005HZ-00 for rddp@ietf.org; Fri, 11 Jul 2003 08:30:07 -0400 Received: from 192.168.0.102 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 69514244A2E; Fri, 11 Jul 2003 08:29:39 -0400 (EDT) Date: Fri, 11 Jul 2003 07:29:34 -0500 From: Caitlin Bestler Subject: Re: [rddp] Review comment on DDP untagged buffer model To: John Hufferd , rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit > I completely agree with Mallikarjun. There are straight forward > techniques to permit the LLP to notice that the buffer is not yet > available, and drop the buffer, thus permitting exploitation of the > basic retry capability of the LLP. On that specific point, I'll agree with you. An incoming packet that lacks a target buffer, but was still an enabled receive, MAY be dropped to allow retransmits. This would conceptually be a feedback from the "placement" interface from the LLP, the DDP would kickback "sorrry, not now". If placement could not be done, completion cannot follow. The LLP would either have to buffer on its own, or drop the packet and rely on the remote peer to retransmit it. However, there should be some caution that this SHOULD NOT be used deliberately as some form of additional flow control. That is the receiving ULP SHOULD strive to always have an adequate buffer capacity for authorized untagged messages. Further, unauthorized untagged messages SHOULD be detected and not only dropped, but result in termination of the DDP stream. We have to be cautious about use of drops on the receiving end being used as a recovery technique. The logical next step is for a receive to decide that tagged buffers don't *really* need to kept in memory, they can be swapped out. When a tagged message arrives we'll just drop it and then start the swap-in process. The other side will recover eventually, and meanwhile I can advertise more buffers than I really own. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Jul 11 12:55:28 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA05889 for ; Fri, 11 Jul 2003 12:55:28 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b1Ak-0002FF-Ri for rddp-archive@odin.ietf.org; Fri, 11 Jul 2003 12:55:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6BGt2mf008628 for rddp-archive@odin.ietf.org; Fri, 11 Jul 2003 12:55:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b1Ak-0002F5-Os for rddp-web-archive@optimus.ietf.org; Fri, 11 Jul 2003 12:55:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA05878 for ; Fri, 11 Jul 2003 12:54:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19b1Ai-0007Ao-00 for rddp-web-archive@ietf.org; Fri, 11 Jul 2003 12:55:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19b1Ai-0007Al-00 for rddp-web-archive@ietf.org; Fri, 11 Jul 2003 12:55:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b1Aj-0002Eq-5a; Fri, 11 Jul 2003 12:55:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b1A4-0002E3-Gk for rddp@optimus.ietf.org; Fri, 11 Jul 2003 12:54:20 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA05848 for ; Fri, 11 Jul 2003 12:54:15 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19b1A2-0007A8-00 for rddp@ietf.org; Fri, 11 Jul 2003 12:54:18 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19b1A2-00079e-00 for rddp@ietf.org; Fri, 11 Jul 2003 12:54:18 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id BADD5245C45; Fri, 11 Jul 2003 12:53:50 -0400 (EDT) Date: Fri, 11 Jul 2003 11:53:45 -0500 From: Caitlin Bestler Subject: Re: [rddp] Review comment on DDP untagged buffer model To: "Mallikarjun C." Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <00a601c34756$9365aea0$66a0f40f@rose.hp.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Thursday, July 10, 2003, cbm@rose.hp.com wrote: > >section 3.2 (Protocol Overview), > " Note, however, that > the ULP needs to address flow control issues because if a DDP > Message arrives for an Untagged Buffer without an associated > receive ULP buffer, the DDP Message is dropped, the DDP Stream > is disabled for reception, and an error is reported to the ULP > at the Data Sink." > >and section 9.1 (Errors detected at the Data Sink). > " 2. The QN and MSN have an associated buffer that allows >Placement > of the payload." > >My suggestion is to change the text along the following lines: >Section 3.2 to: > " Note, however, that the ULP needs to address flow control issues." >and section 9.1 to: > " 2. The QN and MSN have an associated buffer that allows > Placement of the payload. DDP implementations MAY optionally > consider lack of an associated buffer to be not an error, but > such implementations MUST ensure that the DDP Segment is > eventually recovered using LLP means in a ULP-transparent way." As I indicated previously, this is a tricky area. It is very analogous to the problem of the MPA layer detecting a CRC-32c error on material that had already passed the TCP checksum. It is very easy to say, "declare it a bad packet", but that will only work if the retry is initiated at the reverse end above the TCP layer. If your peer's MPA hands TCP segments to a TCP layer cleanly, you will simply get the same faulty TCP segment repeated. The two issues that require care here are: a) The QN/MSN must still be authorized by the ULP. Allowing unauthorized indefinite reception presents major security concerns. Mechanisms that require ULP reaction to being "overlimit" are inherently suspect as methods of preventing a Denial Of Service attack. Forcing frequent trips to the ULP may be a sufficient victory for the attack to succeed. b) While a "conditional placement" interface from the LLP is feasible, and even natural coming from MPA/TCP, it is not something that any ULP should ever rely upon. It is unnatural for the LLP behavior of the receiver, even a rototilled one, to be dependent on higher layers. I won't make that an absolute. I've done network protocol stacks that included the feature that if the processing of a received Ethernet Frame caused an exception the *only* result would be to log the error. Except for the log, the system would simply pretend the packet had never arrived. It was a natural and cheap recovery mechanism that was far preferable to rebooting the machine. I'm not sure what sort of wording, but that is the type of "conditional placement" interface that I think would be acceptable. It is one that *expects* to place the data, or expects to declare the peer to be in violation. "Try again later" should only be reserved as a method of recovering from a system fault. It should never be part of normal operations. When that packet is dropped, the peer LLP is going to assume that it was due to congestion. This will have a major impact. Indeed, when the LLP is SCTP it would be highly incorrect to allow *one* DDP Stream that had under-provisioned its buffers to adversely affect the entire associations congestion window. If that stream really abused the feature, the peer could flip the entire association to an alternate path. An additional factor on this not being something that the ULP should be relying upon is that "conditional placement" is *not* a natural LLP interface for an SCTP stack. Because the SCTP adaptation uses unordered SCTP delivery, the SCTP layer can simply push received Data Chunks to the DDP layer. If the pushed chunk cannot be placed, the DDP layer simply resets the DDP Stream. It has *no* effect on the SCTP layer. Further, this could be cleanly layered over any SCTP stack that had an optimized interface for unordered deliveries. It make senese for the spec to allow the receiving stack some flexibility in handling a *fault* situation. Not having a buffer that should have been present for an authorized untagged message can indeed qualify as a fault. Particularly if the reason it was not available was due to congestion on the system bus or other unusual delays that prevented the ULP from restocking a buffer pool. In response to this type of fault, a false drop is better than a disconnect. But we need to be very clear that the ULP is responsible for properly pacing itself. The Data Sink is expected to be ready to handle all untagged messages that it has authorized reception of. Dropping packets must never be a method of retroactive flow control (oops, I'm busier than I thought I would be). Dropped packets at the transport layer are supposed to represent congestion or transmit errors -- not overworked servers. Most importantly, transport layer drops *will* be interpreted by the peer as congestion or transmit errors. Inducing an incorrect response in your peer is someething that should be avoided, even if recovery will eventually happen. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Jul 11 16:39:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15859 for ; Fri, 11 Jul 2003 16:39:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b4fW-0001o4-Pf for rddp-archive@odin.ietf.org; Fri, 11 Jul 2003 16:39:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6BKd27U006941 for rddp-archive@odin.ietf.org; Fri, 11 Jul 2003 16:39:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b4fW-0001ns-Ji for rddp-web-archive@optimus.ietf.org; Fri, 11 Jul 2003 16:39:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15842 for ; Fri, 11 Jul 2003 16:38:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19b4fU-00022F-00 for rddp-web-archive@ietf.org; Fri, 11 Jul 2003 16:39:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19b4fT-00022C-00 for rddp-web-archive@ietf.org; Fri, 11 Jul 2003 16:38:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b4fU-0001nV-9w; Fri, 11 Jul 2003 16:39:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b4f8-0001n4-Us for rddp@optimus.ietf.org; Fri, 11 Jul 2003 16:38:38 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15822 for ; Fri, 11 Jul 2003 16:38:34 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19b4f6-00021r-00 for rddp@ietf.org; Fri, 11 Jul 2003 16:38:36 -0400 Received: from palrel10.hp.com ([156.153.255.245]) by ietf-mx with esmtp (Exim 4.12) id 19b4f5-00021k-00 for rddp@ietf.org; Fri, 11 Jul 2003 16:38:35 -0400 Received: from rosemail.rose.hp.com (rosemail.rose.hp.com [15.96.64.26]) by palrel10.hp.com (Postfix) with ESMTP id D0CAE1C017F1; Fri, 11 Jul 2003 13:38:28 -0700 (PDT) Received: from swathi (dhcp-roseor-beb139.rose.hp.com [15.23.141.139]) by rosemail.rose.hp.com with SMTP (8.7.1/8.7.3 SMKit7.02) id NAA22770; Fri, 11 Jul 2003 13:38:28 -0700 (PDT) Message-ID: <00f501c347ec$61c7e4d0$66a0f40f@rose.hp.com> From: "Mallikarjun C." To: "Caitlin Bestler" Cc: References: Subject: Re: [rddp] Review comment on DDP untagged buffer model Date: Fri, 11 Jul 2003 13:36:41 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Responding to Caitlin's two posts - > The Data Sink ULP would still be required to authorize each reception Generally, I agree. It may however be the case for some ULPs (such as iSCSI) that they may not be able to precisely predict every possible ULP PDU though the peers know about >95% of the PDUs with certainty (and can provision for them). The ULP in this case can post a few extra "slack" buffers, but it still is theoretically possible in some very corner cases that a local buffer pool may transiently run out. John Hufferd gave an example of processing delays that could also lead to this same scenario. >But I disagree that it is a "benefit" to be forgiving to > remote peers that are sending more untagged messages than they > are allowed. It is actually a performance and security concern. Agreed, I don't think anyone is suggesting being accommodative with bad-behaving remote peers. I am only stating that the connection drop is an RNIC implementation issue, and the DDP spec should leave enough flexibility for implementations here while specifying the precise semantics a DDP implementation should provide to its ULP (RDMAP). >As I indicated previously, this is a tricky area. It is very >analogous to the problem of the MPA layer detecting a CRC-32c >error on material that had already passed the TCP checksum. Not sure I agree that it is "very analogous" though it has some resemblance. In this specific case, assuming the buffers are posted by the retransmit time, an LLP retransmit is guaranteed to recover the packet which isn't the case in what you describe. >b) While a "conditional placement" interface from the LLP >is feasible, and even natural coming from MPA/TCP, it is >not something that any ULP should ever rely upon. No, in fact, the point in my note was that the ULP *must not* be privy to what's going on at the DDP/LLP interface ("in a ULP-transparent way" - my suggested text). In my simplistic analysis, there are two ways that an RNIC might deal with an inbound untagged message that doesn't have an available posted receive buffer. a) RNIC simply drops the connection (as is currently spec'ed in the DDP spec). b) RNIC is "smarter" (or rototilled as protocol designers see it), so drops and recovers the packet later on via LLP means - all the while being transparent to the ULP. My suggestion was that the DDP spec shouldn't mandate either approach. >I'm not sure what sort of wording, but that is the type >of "conditional placement" interface that I think would >be acceptable. Sounds like you're agreeing that it's reasonable to allow either approach (a) or (b) noted above. You however seem to want to specify that allowance in more formal terms than I intended. IMHO, it suffices for the DDP spec to allow implementation flexibility without formalizing the underlying notions. That's what my strawman text changes attempt to do. Regards. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Hewlett-Packard MS 5668 Roseville CA 95747 cbm@rose.hp.com ----- Original Message ----- From: "Caitlin Bestler" To: "Mallikarjun C." Cc: Sent: Thursday, July 10, 2003 8:27 PM Subject: Re: [rddp] Review comment on DDP untagged buffer model > There are two distinct issues when dealing with the Untagged Message > buffer model that must be dealt with seperately: flow control and > provisioning of buffers. > > Requiring the Data Sink to *always* prepost a buffer to enable each > reception of each untagged message is clearly wasteful. I posted an > objection to the last call on the architecture document on this point > earlier. Requiring pre-committed buffers for all eligible untagged > messages simply takes too much memory. A server could easily have > 50,000 clients and allow any one of them to have 6 messages in flight. > > With the simple "one buffer - one authorized MSN" rule you would have > to commit 300,000 buffers. But statistics and/or the aggregate bandwdith > of your connection to those 50,000 clients might mean that only 50,000 > buffers were needed. > > So the protocol needs to allow buffers to be under-provisioned, that is > taken from a common pool, should the local interface choose to offer > such a capability. It certainly would not be a mandatory feature, it > really only applies when you have a large number of connections. > > But there is a second aspect - flow control. > > Each untagged message needs to be authorized by the Data Sink > through some mechanism. Typically this is based upon a session > established number of "send credits". Each untagged message > from the Data Source consumes a credit. A reverse untagged > message typically restores it (although other ULP mechanisms > could be imagined, the NFS drafts allow the credits to be > adjusted by an amount other than one in the reply, for exmaple). > > The question is, who enforces this limit. > > The traditional approach is to require the Data Sink to pre-enable > each receive by posting a buffer. I believe that should be relaxed > only to the extent of eliminating the need for a buffer to be supplied. > The Data Sink ULP would still be required to authorize each reception, > but it MAY ask that the buffer come from some specified pool or even > a system pool (at the discretion of the local interface as to whether > such a resource is available, and if available whether this user > can use it). > > Requiring the ULP to authorize each untagged message in advance > has several advantages: > > -- it allows the receiving endpoint to bound the range of valid > MSNs simply. This range is a very critical resource, the EP > must track the receive status of every message within the > eligible window. If an errant peer jumps ahead 100 MSNs, then > the EP has to track the completion status of all 100 intermediate > messages. > -- If a buffer pool is in fact use, it prevents a single peer from > draining that pool. > > Keep in mind that traditional buffer based flow control does not > apply to RDMA streams. This is dealt with in the applicability > draft in greater detail. But effectively, ULP pacing is more > critical for RDMA applications than for non-RDMA applications > that have the "benefit" of being paced by traditional transport > layer bottlenecks. > > Allowing statistical pooling of buffers is undoubtedly of benefit > to many implementations -- and the protocol should clearly allow > this. But I disagree that it is a "benefit" to be forgiving to > remote peers that are sending more untagged messages than they > are allowed. It is actually a performance and security concern. > >As I indicated previously, this is a tricky area. It is very >analogous to the problem of the MPA layer detecting a CRC-32c >error on material that had already passed the TCP checksum. >It is very easy to say, "declare it a bad packet", but that >will only work if the retry is initiated at the reverse end >above the TCP layer. If your peer's MPA hands TCP segments >to a TCP layer cleanly, you will simply get the same faulty >TCP segment repeated. > >The two issues that require care here are: > >a) The QN/MSN must still be authorized by the ULP. Allowing >unauthorized indefinite reception presents major security >concerns. Mechanisms that require ULP reaction to being >"overlimit" are inherently suspect as methods of preventing >a Denial Of Service attack. Forcing frequent trips to the ULP >may be a sufficient victory for the attack to succeed. > >b) While a "conditional placement" interface from the LLP >is feasible, and even natural coming from MPA/TCP, it is >not something that any ULP should ever rely upon. It is >unnatural for the LLP behavior of the receiver, even a >rototilled one, to be dependent on higher layers. > >I won't make that an absolute. I've done network protocol >stacks that included the feature that if the processing >of a received Ethernet Frame caused an exception the >*only* result would be to log the error. Except for the >log, the system would simply pretend the packet had never >arrived. It was a natural and cheap recovery mechanism >that was far preferable to rebooting the machine. > >I'm not sure what sort of wording, but that is the type >of "conditional placement" interface that I think would >be acceptable. It is one that *expects* to place the data, >or expects to declare the peer to be in violation. "Try >again later" should only be reserved as a method of recovering >from a system fault. It should never be part of normal >operations. > >When that packet is dropped, the peer LLP is going to assume >that it was due to congestion. This will have a major impact. > >Indeed, when the LLP is SCTP it would be highly incorrect to >allow *one* DDP Stream that had under-provisioned its buffers >to adversely affect the entire associations congestion window. >If that stream really abused the feature, the peer could flip >the entire association to an alternate path. > >An additional factor on this not being something that the >ULP should be relying upon is that "conditional placement" >is *not* a natural LLP interface for an SCTP stack. Because >the SCTP adaptation uses unordered SCTP delivery, the SCTP >layer can simply push received Data Chunks to the DDP layer. >If the pushed chunk cannot be placed, the DDP layer simply >resets the DDP Stream. It has *no* effect on the SCTP layer. >Further, this could be cleanly layered over any SCTP stack >that had an optimized interface for unordered deliveries. > >It make senese for the spec to allow the receiving stack some >flexibility in handling a *fault* situation. Not having a buffer >that should have been present for an authorized untagged message >can indeed qualify as a fault. Particularly if the reason it was >not available was due to congestion on the system bus or other >unusual delays that prevented the ULP from restocking a buffer pool. > >In response to this type of fault, a false drop is better than a >disconnect. But we need to be very clear that the ULP is >responsible for properly pacing itself. The Data Sink is >expected to be ready to handle all untagged messages that it has >authorized reception of. Dropping packets must never be a method >of retroactive flow control (oops, I'm busier than I thought I >would be). Dropped packets at the transport layer are supposed >to represent congestion or transmit errors -- not overworked >servers. Most importantly, transport layer drops *will* be >interpreted by the peer as congestion or transmit errors. >Inducing an incorrect response in your peer is someething that >should be avoided, even if recovery will eventually happen. > > > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Jul 11 17:17:28 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA17445 for ; Fri, 11 Jul 2003 17:17:28 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b5GI-0004c4-Ru for rddp-archive@odin.ietf.org; Fri, 11 Jul 2003 17:17:02 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6BLH2Pp017726 for rddp-archive@odin.ietf.org; Fri, 11 Jul 2003 17:17:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b5GI-0004bp-Ot for rddp-web-archive@optimus.ietf.org; Fri, 11 Jul 2003 17:17:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA17426 for ; Fri, 11 Jul 2003 17:16:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19b5GG-0002ZQ-00 for rddp-web-archive@ietf.org; Fri, 11 Jul 2003 17:17:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19b5GF-0002ZN-00 for rddp-web-archive@ietf.org; Fri, 11 Jul 2003 17:16:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b5GH-0004bQ-Dn; Fri, 11 Jul 2003 17:17:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b5Fm-0004am-8C for rddp@optimus.ietf.org; Fri, 11 Jul 2003 17:16:30 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA17401 for ; Fri, 11 Jul 2003 17:16:25 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19b5Fj-0002Z5-00 for rddp@ietf.org; Fri, 11 Jul 2003 17:16:27 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19b5Fj-0002YU-00 for rddp@ietf.org; Fri, 11 Jul 2003 17:16:27 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 81D0F2451B1; Fri, 11 Jul 2003 17:16:00 -0400 (EDT) Date: Fri, 11 Jul 2003 16:15:55 -0500 From: Caitlin Bestler Subject: Re: [rddp] Review comment on DDP untagged buffer model To: "Mallikarjun C." Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <00f501c347ec$61c7e4d0$66a0f40f@rose.hp.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Friday, July 11, 2003, cbm@rose.hp.com wrote: > >b) While a "conditional placement" interface from the LLP >>is feasible, and even natural coming from MPA/TCP, it is >>not something that any ULP should ever rely upon. > >No, in fact, the point in my note was that the ULP *must not* >be privy to what's going on at the DDP/LLP interface ("in a >ULP-transparent way" - my suggested text). > >In my simplistic analysis, there are two ways that an RNIC might >deal with an inbound untagged message that doesn't have an >available posted receive buffer. > >a) RNIC simply drops the connection (as is currently spec'ed in the > DDP spec). > >b) RNIC is "smarter" (or rototilled as protocol designers see it), so > drops and recovers the packet later on via LLP means - > all the while being transparent to the ULP. > >My suggestion was that the DDP spec shouldn't mandate either >approach. What I meant by conditional placement being something that the ULP never relies upon, is that the ULP should never use the feeding of buffers independenlty of authorizing receives as a means of flow control. So a ULP should never authorize 100 receives, and then only supply 50 buffers if the purpose is to *stall* the 51st and later responses. If the ULP is supplying only 50 buffers it is because it does not expect their to be a 51st message. Having a nice recovery mechanism in case that estimate was off is certainly nice, but what I think we want to avoid is the ULP *counting* on the lack of buffers to act as a throttle. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Jul 11 17:21:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA17583 for ; Fri, 11 Jul 2003 17:21:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b5KB-0004jM-Ix for rddp-archive@odin.ietf.org; Fri, 11 Jul 2003 17:21:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6BLL3Cd018178 for rddp-archive@odin.ietf.org; Fri, 11 Jul 2003 17:21:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b5KB-0004j7-Dh for rddp-web-archive@optimus.ietf.org; Fri, 11 Jul 2003 17:21:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA17563 for ; Fri, 11 Jul 2003 17:20:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19b5K9-0002cM-00 for rddp-web-archive@ietf.org; Fri, 11 Jul 2003 17:21:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19b5K8-0002cJ-00 for rddp-web-archive@ietf.org; Fri, 11 Jul 2003 17:21:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b5KA-0004iU-6c; Fri, 11 Jul 2003 17:21:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b5JI-0004fv-3n for rddp@optimus.ietf.org; Fri, 11 Jul 2003 17:20:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA17534 for ; Fri, 11 Jul 2003 17:20:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19b5JF-0002bf-00 for rddp@ietf.org; Fri, 11 Jul 2003 17:20:05 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19b5JF-0002bM-00 for rddp@ietf.org; Fri, 11 Jul 2003 17:20:05 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 6BCDA245235; Fri, 11 Jul 2003 17:19:38 -0400 (EDT) Date: Fri, 11 Jul 2003 16:19:33 -0500 From: Caitlin Bestler Subject: Re: [rddp] Review comment on DDP untagged buffer model To: "Mallikarjun C." Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <00f501c347ec$61c7e4d0$66a0f40f@rose.hp.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Friday, July 11, 2003, cbm@rose.hp.com wrote: >>But I disagree that it is a "benefit" to be forgiving to >> remote peers that are sending more untagged messages than they >> are allowed. It is actually a performance and security concern. > >Agreed, I don't think anyone is suggesting being accommodative >with bad-behaving remote peers. I am only stating that the >connection drop is an RNIC implementation issue, and the DDP >spec should leave enough flexibility for implementations here while >specifying the precise semantics a DDP implementation should >provide to its ULP (RDMAP). I don't think there's a disagreement here, but my point is that terminating the stream for non-authorized untagged messages is feature of the protocol, not an implementation detail. The handling of an authorized untagged message that is unfortunately missing a buffer is what can be considered an implementation detail. The implementation MAY terminate the connection, or it MAY do something smarter like wait for a retransmit. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Jul 11 19:34:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA24578 for ; Fri, 11 Jul 2003 19:34:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b7Ou-0004l5-6i for rddp-archive@odin.ietf.org; Fri, 11 Jul 2003 19:34:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6BNY4PW018289 for rddp-archive@odin.ietf.org; Fri, 11 Jul 2003 19:34:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b7Ou-0004ku-3B for rddp-web-archive@optimus.ietf.org; Fri, 11 Jul 2003 19:34:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA24557 for ; Fri, 11 Jul 2003 19:33:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19b7Os-0004mF-00 for rddp-web-archive@ietf.org; Fri, 11 Jul 2003 19:34:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19b7Or-0004mC-00 for rddp-web-archive@ietf.org; Fri, 11 Jul 2003 19:34:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b7Or-0004kQ-Id; Fri, 11 Jul 2003 19:34:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19b7OI-0004jw-FN for rddp@optimus.ietf.org; Fri, 11 Jul 2003 19:33:26 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA24527 for ; Fri, 11 Jul 2003 19:33:21 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19b7OG-0004lX-00 for rddp@ietf.org; Fri, 11 Jul 2003 19:33:24 -0400 Received: from atlrel9.hp.com ([156.153.255.214]) by ietf-mx with esmtp (Exim 4.12) id 19b7OF-0004lT-00 for rddp@ietf.org; Fri, 11 Jul 2003 19:33:23 -0400 Received: from rosemail.rose.hp.com (rosemail.rose.hp.com [15.96.64.26]) by atlrel9.hp.com (Postfix) with ESMTP id D06CA1C0184E; Fri, 11 Jul 2003 19:33:22 -0400 (EDT) Received: from swathi (dhcp-roseor-beb139.rose.hp.com [15.23.141.139]) by rosemail.rose.hp.com with SMTP (8.7.1/8.7.3 SMKit7.02) id QAA04747; Fri, 11 Jul 2003 16:33:21 -0700 (PDT) Message-ID: <012001c34804$d07407c0$66a0f40f@rose.hp.com> From: "Mallikarjun C." To: Cc: Date: Fri, 11 Jul 2003 16:30:34 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Content-Transfer-Encoding: 7bit Subject: [rddp] Re: CRC Rough Consensus recap Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit David, Can you please provide a couple of clarifications on two concerns I have? Apologies in advance for the somewhat late request, and also if there is some other relevant discussion that I haven't followed (I admit I didn't closely follow them all) here. Based on some early discussion, I had assumed that the ULP designers can safely assume a CRC-level data integrity from an RDMAP/DDP/MPA/TCP stack. In reading the A-E' calls (that is, if I understand them as intended), I am seeing two aspects that somewhat concern me because these doesn't seem to assure ULPs in making that safe assumption. a) The "not visible to ULPs" wording in E'. I understand the administrative configuration part and the discussion that went into it. Can you please confirm that despite this wording, ULP designers will be on a sure footing in assuming a CRC-level integrity? b) Even while stating that the CRC implementation is a MUST, this set of consensus calls is clearly allowing the implementations to drop the connection ("break the connection if the different preference have created a situation from which it is impossible to continue") if the other side forces the usage of MPA-CRC . IOW, as I read this, there's no such thing as a "default" or guaranteed CRC-supported operation anymore (as I had originally understood the early discussion). Can you please confirm that this text again doesn't preclude the ULP designers in making that safe assumption? [ i.e. the side that didn't want MPA CRC chose so only because it knew that there's a comparable level of assured data integrity for the data path ] Thanks. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Hewlett-Packard MS 5668 Roseville CA 95747 cbm@rose.hp.com >To: rddp@ietf.org >Subject: [rddp] CRC Rough Consensus recap >From: Black_David@emc.com >Date: Wed, 9 Jul 2003 19:58:39 -0400 >List-help: >List-id: IETF Remote Direct Data Placement (rddp) WG >List-post: >List-subscribe: , >List-unsubscribe: , >Sender: rddp-admin@ietf.org > >-------------------------------------------------------------------------------- > >I thought I sent this out to the list a week or so ago, >but some helpful mailer seems to have found it entirely >too tasty ... > >I believe the following items represent the rough consensus >of the RDDP WG on CRC requirements for the TCP mapping >of the RDDP protocols: > >(A) CRC support MUST be implemented. > >(B) CRC usage is all-or-nothing on header and data. > There is no requirement for CRC on header but > not on data or CRC on data but not on header. > >NB: This does not imply that there is a single CRC > vs. split header and data CRCs, as there are some > additional issues bearing on this. > >(C) CRC usage is bidirectional. Either CRCs are used > in both directions on a connection, or in neither. > There is no requirement for CRC usage in one > direction and not the other. > >(D) CRC omission requires consent of both ends of the > connection. If either end of the connection > specifies that CRCs are to be used, then they > MUST be used [in both directions (C)]. This entails > some sort of mechanism to enforce CRC usage when > the two ends of the connection start out with > different preferences (or break the connection > if the different preference have created a > situation from which it is impossible to continue). > >(E) CRCs are OPTIONAL to use, but (per (D) above), > either end of the connection can force CRC > usage in both directions. > >(E') CRC usage is an administrative configuration decision > - it is not intended to be visible to ULPs. > >This represents an affirmation of the consensus calls on >(B), (E) and (E'), as I have seen no objections to them >since they were posted. > >As noted earlier, there has been considerable elaboration >on (E') - the right place to capture that elaboration and >tell ULP implementers what they should expect and assume is >the applicability statement (draft-ietf-rddp-applicability-00.txt). >Please keep this in mind in reviewing that draft for discussion >in Vienna. > >Thanks, >--David > >---------------------------------------------------- >David L. Black, Senior Technologist >EMC Corporation, 176 South St., Hopkinton, MA 01748 >+1 (508) 293-7953 FAX: +1 (508) 293-7786 >black_david@emc.com Mobile: +1 (978) 394-7754 >---------------------------------------------------- > > > -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Hewlett-Packard MS 5668 Roseville CA 95747 cbm@rose.hp.com _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Sat Jul 12 00:49:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA08264 for ; Sat, 12 Jul 2003 00:49:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19bCJk-0006ZM-VW for rddp-archive@odin.ietf.org; Sat, 12 Jul 2003 00:49:05 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6C4n4MG025246 for rddp-archive@odin.ietf.org; Sat, 12 Jul 2003 00:49:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19bCJk-0006Z7-S9 for rddp-web-archive@optimus.ietf.org; Sat, 12 Jul 2003 00:49:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA08223 for ; Sat, 12 Jul 2003 00:48:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19bCJi-0000c1-00 for rddp-web-archive@ietf.org; Sat, 12 Jul 2003 00:49:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19bCJg-0000bx-00 for rddp-web-archive@ietf.org; Sat, 12 Jul 2003 00:49:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19bCJh-0006Yn-AC; Sat, 12 Jul 2003 00:49:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19bCIy-0006YM-Fc for rddp@optimus.ietf.org; Sat, 12 Jul 2003 00:48:16 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA08172 for ; Sat, 12 Jul 2003 00:48:11 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19bCIv-0000b7-00 for rddp@ietf.org; Sat, 12 Jul 2003 00:48:13 -0400 Received: from rwcrmhc12.comcast.net ([216.148.227.85]) by ietf-mx with esmtp (Exim 4.12) id 19bCIv-0000aX-00 for rddp@ietf.org; Sat, 12 Jul 2003 00:48:13 -0400 Received: from cs.uchicago.edu (h005018030f43.ne.client2.attbi.com[24.128.140.161](untrusted sender)) by attbi.com (rwcrmhc12) with SMTP id <20030712044743014003oj79e>; Sat, 12 Jul 2003 04:47:43 +0000 To: rddp@ietf.org Subject: Re: [rddp] Review comment on DDP untagged buffer model In-Reply-To: Message from Caitlin Bestler of "Thu, 10 Jul 2003 22:27:03 CDT." References: Date: Sat, 12 Jul 2003 00:47:39 -0400 From: Stephen Bailey Message-Id: Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Caitlin, Sorry I've been slow responding to your previous message on this topic. I'll roll it into this message. The executive summary is that you should be picking these bones with the protocols, not the architecture. I think the first paragraph of the architecture document makes this clear. > There are two distinct issues when dealing with the Untagged Message > buffer model that must be dealt with seperately: flow control and > provisioning of buffers. The architecture IS NOT defining an interface (API, or a looser Verbs notion, or whatever else) for RDDP. It's only modelling what the protocols do, in terms of messages and a set of endpoint behavior constraints. The C notation is a terse way to express the model. Don't read too much into it. The RDDP architecture does not concern itself with `provisioning of buffers' for untagged messages. Untagged messages are, well, messages: atomically sent and delivered blobs of data. The buffer abstraction in the RDDP architecture is only for tagged messages. How you implement receiving of untagged messages are implementation choice. As long as an implemention respects the relevant ordering requirements, you can do it any way you want (shared pools, unshared pools, zero copy, zillion copy, very large integers, light pulses, DNA molecules, whatever). The receive operation's (gimmie the next message or notification) role in the architecture is to express that DDP and RDMA allow untagged message transfer, and that there are defined sequencing rules for buffer operations (set()s, get()s and notifications) against ULP `blob gimmie' operations. The protocols nail down the sequencing rules. The architecture just says the exist (i.e. sequencing is a central concept). The architecture's also silent on untagged message flow control. The protocols are where you'd say something (and argue) about flow control. To repeat, the architecture document IS NOT a specification. It's just a model that frames what the specifications describe (as in `what the hell are we talking about with this DDP and RDMA jazz anyway?') If you're saying there should be an explicit model of untagged message buffering in the architecture (forcing that concept into the protocols as well), I'm saying no there shouldn't be. Thanks, Steph _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Sat Jul 12 14:00:32 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA00268 for ; Sat, 12 Jul 2003 14:00:32 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19bOfE-0007mb-PC for rddp-archive@odin.ietf.org; Sat, 12 Jul 2003 14:00:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6CI04Fh029905 for rddp-archive@odin.ietf.org; Sat, 12 Jul 2003 14:00:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19bOfE-0007mG-5l for rddp-web-archive@optimus.ietf.org; Sat, 12 Jul 2003 14:00:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA00229 for ; Sat, 12 Jul 2003 14:00:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19bOfB-00030b-00 for rddp-web-archive@ietf.org; Sat, 12 Jul 2003 14:00:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19bOfB-00030Y-00 for rddp-web-archive@ietf.org; Sat, 12 Jul 2003 14:00:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19bOfB-0007lu-Ed; Sat, 12 Jul 2003 14:00:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19bOeK-0007lQ-KE for rddp@optimus.ietf.org; Sat, 12 Jul 2003 13:59:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA00181 for ; Sat, 12 Jul 2003 13:59:05 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19bOeI-0002zw-00 for rddp@ietf.org; Sat, 12 Jul 2003 13:59:06 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19bOeG-0002z5-00 for rddp@ietf.org; Sat, 12 Jul 2003 13:59:05 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 99693244F28; Sat, 12 Jul 2003 13:58:34 -0400 (EDT) Date: Sat, 12 Jul 2003 12:58:26 -0500 From: Caitlin Bestler To: Stephen Bailey , rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] Some Minimal end-to-end Buffer Rules Required Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Saturday, July 12, 2003, steph@cs.uchicago.edu wrote: >Caitlin, > >Sorry I've been slow responding to your previous message on this >topic. I'll roll it into this message. > >The executive summary is that you should be picking these bones with >the protocols, not the architecture. I think the first paragraph of >the architecture document makes this clear. > >> There are two distinct issues when dealing with the Untagged Message >> buffer model that must be dealt with seperately: flow control and >> provisioning of buffers. > >The architecture IS NOT defining an interface (API, or a looser Verbs >notion, or whatever else) for RDDP. It's only modelling what the >protocols do, in terms of messages and a set of endpoint behavior >constraints. The C notation is a terse way to express the model. >Don't read too much into it. > Perhaps it does not belong in the architecture draft, but that's where it was in the earlier drafts. The first draft required the local ULP to authorize untagged reception by posting a buffer. That was reduced to merely requiring the ULP to authorize untagged reception, but being silent about where buffers came from, and then to saying nothing about how untagged receptions are regulated. But the end-to-end protocol needs for certain rules about buffer usage to be explicitly stated. I suppose we can debate which draft they belong in precisely, but that does not mean these issues can be left totally to each implementation. There are certain required end-to-end behaviors needed to allow ULPs to work. I don't believe there is any debate on tagged buffers: -- Tagged messages may only access local memory to the extent that is explicitly authorized by the local ULP. -- The local interface MUST allow the local ULP to enable any contiguous subset of its address space through an STag. -- The local interface MAY allow other combinations of ULP accessible memory to be enabled for access through an STag. For example, zero-based buffers or maps formed from physical buffer lists. -- The local interface MUST NOT allow the ULP to enable an STag with memory access rights that the ULP does not itself already have. -- The remote peer MUST NOT use an advertised buffer in any way that is not explicitly authorized by the Upper Layer Protocol. -- The local ULP MUST use the local interface to enable STags for all access rights advertised to the remote peer. It MAY enable the STags for more rights, but SHOULD NOT do so if that places any other process or client at risk. -- The DDP Layer MUST terminate a DDP Stream if an incoming DDP Segment attempts to use an STag in a way that was not enabled by the local ULP. I believe the following rules are required for untagged buffers -- Untagged messages may only be sent as explicitly allowed by the Upper Layer Protocol. The ULP MUST define some mechanism for granting permission to send Untagged messages. -- Authorization for an untagged message MUST specify a maximum message size. Receipt of an untagged message that is larger MUST result in termination of the DDP Stream. -- Authorized untagged messages SHOULD be placed into a ULP accessible buffer. The local interface MUST allow the ULP to supply its own buffer for each authorized untagged message. The local interface MAY alternately allow the buffers to be filled from a shared resource. -- Unauthorized untagged messages SHOULD terminate the DDP Stream. Failure to terminate an "over credit" DDP Stream MUST NOT imperil the availability of resources for any compliant DDP Stream. -- The receiving side SHOULD have an adequate supply of buffers to handle all authorized untagged messages. However, this does not require pre-committment of one buffer for each authorized untagged message. Statistical sharing of buffer resources MAY be used. However the receiving side MUST NOT use this as a form of ex post facto flow control. With these rules, iWARP has extremely powerful and flexible flow control that is directly responsive to actual ULP needs. It is superior to the flow control available to non-RDMA IP ULPs. However, without these rules, an iWARP ULP is both more fragile and more of a security risk than a conventional non-RDMA ULP. There are several critical reasons for this: -- Transport layer buffers no longer pace the stream A principal benefit of RDMA protocols is eliminating dependence on transport layer buffering. But it is easy to overlook the fact that as hated as any bottleneck is, they also have side-effects that may have been beneficial. The side effect of being forced to pass-through the narrow capacity of transport layer buffering was some flow control. With an RDMA protocol there is no predicting where the next segment to arrive will be placed. Therefore it would be incorrect to hold it off because *one* of the logical destinations may have exhausted its capacity. The optimal interface between the LLP and DDP allows the LLP to pass each validated chunk/segment/whatever to the DDP as soon as it is validated and say, "here put this wherever it goes". Such an interface, if unchecked, would allow a remote peer to rapidly consume available buffers. -- The LLP cannot detect violations by the remote peer All DDP Segments look the same to the LLP. -- The ULP cannot detect violations by the remote peer Virtually all local interfaces that support RDMA protocols use a completion queue to deliver completions of untagged messages. Frequently these completion queues are: * Opaque. The ULP cannot knows the contents of the queue except by dequeuing them serially. * Shared across multiple DDP Streams. The net effect is that the ULP cannot know when an untagged message was received. By the time the completion is reaped, an indeterminant amount of time will have passed. The remote peer may have been granted permission during that time. Placing responsiblity for catching "cheaters" who sent one or two "extra" untagged messages early on the ULP will virtually guarantee that cheating will not be caught. If clients are competing for resources, for example when they are all using a common file server, it would be to the each of the clients advantage to cheat. Or at the minimum, because they are competing for resources, any client that has an "off by one" error will be far more motivated to fix it if they are sending one *too few* requests to the server. This could easily escalate into an arms race, where everyone assumes that the Server is counting on the clients cheating by one, and that playing by the rules would place the client at a competitive disadvantage versus other clients. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 03:31:40 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA29617 for ; Mon, 14 Jul 2003 03:31:40 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19bxnj-0000DL-VJ for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 03:31:12 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6E7VBfR000823 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 03:31:11 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19bxnj-0000DC-QZ for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 03:31:11 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA29598 for ; Mon, 14 Jul 2003 03:31:09 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19bxnh-0002Jm-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 03:31:09 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19bxng-0002Ji-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 03:31:08 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19bxna-0000BH-WF; Mon, 14 Jul 2003 03:31:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19bxnL-00006d-Fp for rddp@optimus.ietf.org; Mon, 14 Jul 2003 03:30:48 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA29582 for ; Mon, 14 Jul 2003 03:30:45 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19bxnJ-0002Jd-00 for rddp@ietf.org; Mon, 14 Jul 2003 03:30:45 -0400 Received: from mxic2.corp.emc.com ([128.221.31.40]) by ietf-mx with esmtp (Exim 4.12) id 19bxnI-0002JA-00 for rddp@ietf.org; Mon, 14 Jul 2003 03:30:44 -0400 Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Mon, 14 Jul 2003 03:30:08 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0C2748DA@corpmx14.us.dg.com> To: cbm@rose.hp.com, rddp@ietf.org Subject: RE: [rddp] Review comment on DDP untagged buffer model Date: Mon, 14 Jul 2003 03:20:18 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , On this topic, starting with Mallikarjun's original message: > The DDP spec implies that the DDP Stream is to be > dropped by implementations when there is no receive > buffer to place an inbound untagged message. This > is mandating a specific type of implementation, and IMHO, > is needless. The DDP spec should instead allow for > implementation flexibility in this area. Those RNICs > that can gracefully deal with lack of receive buffers > via LLP means (such as TCP retransmit) should be > allowed as well. I can agree with the authors' original intent > that ULP-level flow control and/or adequate buffer provisioning > must be done to avoid needless retransmits, but mandating > a connection drop on the first instance of hitting a (most > likely a very transient) no-buffer window is severe. plus one of Caitlin's comments that is very much on target: > It make senese for the spec to allow the receiving stack some > flexibility in handling a *fault* situation. Not having a buffer > that should have been present for an authorized untagged message > can indeed qualify as a fault. I suggest that treating this sort of situation as a "fault" is a good way to think about it - among the reasons for this is that if TCP retransmit is the recovery mechanism, frequent and/or routine usage of this sort of recovery has the potential to run afoul of the "SHOULD be capable of queuing out-of-order TCP segments" requirement in Section 4.2.2.20 of RFC 1122. Mallikarjun's proposed text revisions don't seem to capture the sense that this situation is a "fault" that SHOULD NOT be permitted to occur repeatedly or frequently. I agree that the current requirement "is severe", but OTOH implementations cannot rely on LLP retransmit or the use of "hidden" buffers of some form to recover from this entire class of situations. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 03:46:34 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA29931 for ; Mon, 14 Jul 2003 03:46:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19by29-0000yU-Te for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 03:46:05 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6E7k5pg003738 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 03:46:05 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19by27-0000yB-CT for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 03:46:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA29889 for ; Mon, 14 Jul 2003 03:46:01 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19by24-0002OD-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 03:46:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19by24-0002OA-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 03:46:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19by25-0000xi-Pz; Mon, 14 Jul 2003 03:46:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19by1I-0000wo-2R for rddp@optimus.ietf.org; Mon, 14 Jul 2003 03:45:12 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id DAA29883 for ; Mon, 14 Jul 2003 03:45:09 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19by1F-0002O1-00 for rddp@ietf.org; Mon, 14 Jul 2003 03:45:09 -0400 Received: from [168.159.2.8] (helo=mailhub.lss.emc.com) by ietf-mx with esmtp (Exim 4.12) id 19by1E-0002Ny-00 for rddp@ietf.org; Mon, 14 Jul 2003 03:45:08 -0400 Received: from mxic1.corp.emc.com (mxic1.isus.emc.com [168.159.129.100]) by mailhub.lss.emc.com (Switch-2.2.5/Switch-2.2.0) with ESMTP id h6E7U6o12034; Mon, 14 Jul 2003 03:30:07 -0400 (EDT) Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Mon, 14 Jul 2003 03:30:06 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0C2748D9@corpmx14.us.dg.com> To: cbm@rose.hp.com Cc: rddp@ietf.org Date: Mon, 14 Jul 2003 03:20:17 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] RE: CRC Rough Consensus recap Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Mallikarjun, > Can you please provide a couple of clarifications on two > concerns I have? Sure, [... snip ...] > a) The "not visible to ULPs" wording in E'. I understand > the administrative configuration part and the discussion that > went into it. Can you please confirm that despite this wording, > ULP designers will be on a sure footing in assuming a CRC-level > integrity? (E') was not the end of the discussion, as there has been subsequent elaboration. In any case, I believe the answer is yes and (as indicated in my original message) this should be stated and explained clearly in the applicability draft -- the place to tell ULP designers what to expect/assume about RDDP. > b) Even while stating that the CRC implementation is a MUST, > this set of consensus calls is clearly allowing the implementations > to drop the connection ("break the connection if the different > preference have created a situation from which it is impossible > to continue") if the other side forces the usage of MPA-CRC . If an implementation always does this, the implementation violates (D)'s requirement that both ends of the connection consent to CRC omission (in essence the implementation is saying "I will only talk to you if you agree to omit CRCs", which is not permitted by (D) ). **If** the mechanism that realizes (D)'s requirements on CRC usage has cases in which the connection must be dropped due to incompatible CRC usage preferences, then a requirement is needed to cause some subsequent connection attempt to succeed - a likely way to accomplish this would be to force both sides to express a preference for CRCs on retry when a connection is dropped due to incompatible CRC usage preference (but note the "**If**" above). I think forcing CRC usage when there's doubt is safer than trying to figure out when to believe the other end's view of connection quality. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- > -----Original Message----- > From: Mallikarjun C. [mailto:cbm@rose.hp.com] > Sent: Friday, July 11, 2003 7:31 PM > To: Black_David@emc.com > Cc: rddp@ietf.org > Subject: Re: CRC Rough Consensus recap > > > David, > > Can you please provide a couple of clarifications on two > concerns I have? Apologies in advance for the somewhat > late request, and also if there is some other relevant discussion > that I haven't followed (I admit I didn't closely follow them all) > here. > > Based on some early discussion, I had assumed that the > ULP designers can safely assume a CRC-level data integrity > from an RDMAP/DDP/MPA/TCP stack. In reading the A-E' > calls (that is, if I understand them as intended), I am seeing two > aspects that somewhat concern me because these doesn't seem > to assure ULPs in making that safe assumption. > > a) The "not visible to ULPs" wording in E'. I understand > the administrative configuration part and the discussion that > went into it. Can you please confirm that despite this wording, > ULP designers will be on a sure footing in assuming a CRC-level > integrity? > > b) Even while stating that the CRC implementation is a MUST, > this set of consensus calls is clearly allowing the implementations > to drop the connection ("break the connection if the different > preference have created a situation from which it is impossible > to continue") if the other side forces the usage of MPA-CRC . > IOW, as I read this, there's no such thing as a "default" or guaranteed > CRC-supported operation anymore (as I had originally understood > the early discussion). Can you please confirm that this text again > doesn't preclude the ULP designers in making that safe assumption? > [ i.e. the side that didn't want MPA CRC chose so only because > it knew that there's a comparable level of assured data integrity > for the data path ] > > Thanks. > -- > Mallikarjun > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions > Hewlett-Packard MS 5668 > Roseville CA 95747 > cbm@rose.hp.com > > > >To: rddp@ietf.org > >Subject: [rddp] CRC Rough Consensus recap > >From: Black_David@emc.com > >Date: Wed, 9 Jul 2003 19:58:39 -0400 > >List-help: > >List-id: IETF Remote Direct Data Placement (rddp) WG > >List-post: > >List-subscribe: > , uest@ietf.org?subject=subscribe> > >List-unsubscribe: > , uest@ietf.org?subject=unsubscribe> > >Sender: rddp-admin@ietf.org > > > >------------------------------------------------------------- > ------------------- > > > >I thought I sent this out to the list a week or so ago, > >but some helpful mailer seems to have found it entirely > >too tasty ... > > > >I believe the following items represent the rough consensus > >of the RDDP WG on CRC requirements for the TCP mapping > >of the RDDP protocols: > > > >(A) CRC support MUST be implemented. > > > >(B) CRC usage is all-or-nothing on header and data. > > There is no requirement for CRC on header but > > not on data or CRC on data but not on header. > > > >NB: This does not imply that there is a single CRC > > vs. split header and data CRCs, as there are some > > additional issues bearing on this. > > > >(C) CRC usage is bidirectional. Either CRCs are used > > in both directions on a connection, or in neither. > > There is no requirement for CRC usage in one > > direction and not the other. > > > >(D) CRC omission requires consent of both ends of the > > connection. If either end of the connection > > specifies that CRCs are to be used, then they > > MUST be used [in both directions (C)]. This entails > > some sort of mechanism to enforce CRC usage when > > the two ends of the connection start out with > > different preferences (or break the connection > > if the different preference have created a > > situation from which it is impossible to continue). > > > >(E) CRCs are OPTIONAL to use, but (per (D) above), > > either end of the connection can force CRC > > usage in both directions. > > > >(E') CRC usage is an administrative configuration decision > > - it is not intended to be visible to ULPs. > > > >This represents an affirmation of the consensus calls on > >(B), (E) and (E'), as I have seen no objections to them > >since they were posted. > > > >As noted earlier, there has been considerable elaboration > >on (E') - the right place to capture that elaboration and > >tell ULP implementers what they should expect and assume is > >the applicability statement (draft-ietf-rddp-applicability-00.txt). > >Please keep this in mind in reviewing that draft for discussion > >in Vienna. > > > >Thanks, > >--David > > > >---------------------------------------------------- > >David L. Black, Senior Technologist > >EMC Corporation, 176 South St., Hopkinton, MA 01748 > >+1 (508) 293-7953 FAX: +1 (508) 293-7786 > >black_david@emc.com Mobile: +1 (978) 394-7754 > >---------------------------------------------------- > > > > > > > > > > -- > Mallikarjun > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions > Hewlett-Packard MS 5668 > Roseville CA 95747 > cbm@rose.hp.com _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 05:57:36 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA04449 for ; Mon, 14 Jul 2003 05:57:36 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c04z-0006zy-Mu for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 05:57:10 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6E9v9YF026899 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 05:57:09 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c04z-0006zm-D9 for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 05:57:09 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA04410 for ; Mon, 14 Jul 2003 05:57:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c04r-0003WT-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 05:57:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19c04q-0003WQ-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 05:57:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c04q-0006zE-LF; Mon, 14 Jul 2003 05:57:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c04o-0006yo-9t for rddp@optimus.ietf.org; Mon, 14 Jul 2003 05:56:58 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA04400 for ; Mon, 14 Jul 2003 05:56:54 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c04k-0003W7-00 for rddp@ietf.org; Mon, 14 Jul 2003 05:56:54 -0400 Received: from e34.co.us.ibm.com ([32.97.110.132]) by ietf-mx with esmtp (Exim 4.12) id 19c04j-0003VN-00 for rddp@ietf.org; Mon, 14 Jul 2003 05:56:53 -0400 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e34.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h6E9uDEh255364; Mon, 14 Jul 2003 05:56:13 -0400 Received: from d03nm115.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay02.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h6E9uAuX072020; Mon, 14 Jul 2003 03:56:10 -0600 To: Black_David@emc.com Cc: cbm@rose.hp.com, rddp@ietf.org, rddp-admin@ietf.org MIME-Version: 1.0 Subject: RE: [rddp] Review comment on DDP untagged buffer model X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 From: John Hufferd Message-ID: Date: Mon, 14 Jul 2003 02:56:10 -0700 X-MIMETrack: Serialize by Router on D03NM115/03/M/IBM(Release 6.0.1 [IBM]|June 10, 2003) at 07/14/2003 03:56:12, Serialize complete at 07/14/2003 03:56:12 Content-Type: multipart/alternative; boundary="=_alternative 00368EF188256D63_=" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multipart message in MIME format. --=_alternative 00368EF188256D63_= Content-Type: text/plain; charset="us-ascii" David, In your response that said: "I agree that the current requirement "is severe", but OTOH implementations cannot rely on LLP retransmit or the use of "hidden" buffers of some form to recover from this entire class of situations.". This response caused me to pause and re-read it a number of times. At the conclusion of the re-reading I am not sure that I agree or disagree. This is because the missing buffer situation can be caused by things out of the control of the ULP. And therefore not be a ULP error. Suppose we have a application that manages many (perhaps thousands of connections) and therefore needs to manage a buffer pool and queue the approprate number of buffers (WQEs etc.) at various times. There are clearly ways to determine, based on the link speed, the maximum number of buffers to queue, and at what interval to replenish the queue. However, it is also possible that the OS many have some high priority system task which gets invoked just as the ULP is about to post additional buffers. In this case it is possible that authorized responses from the various remote nodes may use up the posted buffers, before the ULP can get control and post additional ones. This is an important situation since even though the various remote nodes are given authorization (credits) to send in a message they may wait for some unknown time before sending the message and thus using the credit. Since we can not determine when the message will be sent, the ULP will need to manage a buffer pool. And the only thing it knows is the all the combined remote nodes can not exceed the bandwidth of the connection, and therefore upper buffer limit can be established. When you take this upper limit and add some number of extra buffers, to handle the most probable cases of high priority system task getting in the way, you have a reasonable approach for managing a buffer pool. However, some system vendors would like to protect customers from what ever impact might be caused by the system itself or other higher priority tasks getting in the way of the ULP's buffer replenishment. (Especially since the connection, which may be struck by this problem is not predictable since it may be one of thousands that are sharing the buffer pool.) Therefore, these systems vendors would like to pick a RNIC vendor that did not cause ULP connections to be ended because of some higher priority tasks getting in the way of the buffer replenishment. I feel therefore, that having a HARD requirement that the connection MUST be terminated if a buffer is not available, is not the right thing to do. So I am not OK with treating a buffer shortage for receiving an "RDMA Send" messages as an error that must terminate the connection. But I can not determine from your response whether you agree or not. I am just not sure what words you are suggesting that we include. . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/System Group, San Jose CA Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Alt Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com Black_David@emc.com Sent by: rddp-admin@ietf.org 07/14/2003 12:20 AM To: cbm@rose.hp.com, rddp@ietf.org cc: Subject: RE: [rddp] Review comment on DDP untagged buffer model On this topic, starting with Mallikarjun's original message: > The DDP spec implies that the DDP Stream is to be > dropped by implementations when there is no receive > buffer to place an inbound untagged message. This > is mandating a specific type of implementation, and IMHO, > is needless. The DDP spec should instead allow for > implementation flexibility in this area. Those RNICs > that can gracefully deal with lack of receive buffers > via LLP means (such as TCP retransmit) should be > allowed as well. I can agree with the authors' original intent > that ULP-level flow control and/or adequate buffer provisioning > must be done to avoid needless retransmits, but mandating > a connection drop on the first instance of hitting a (most > likely a very transient) no-buffer window is severe. plus one of Caitlin's comments that is very much on target: > It make senese for the spec to allow the receiving stack some > flexibility in handling a *fault* situation. Not having a buffer > that should have been present for an authorized untagged message > can indeed qualify as a fault. I suggest that treating this sort of situation as a "fault" is a good way to think about it - among the reasons for this is that if TCP retransmit is the recovery mechanism, frequent and/or routine usage of this sort of recovery has the potential to run afoul of the "SHOULD be capable of queuing out-of-order TCP segments" requirement in Section 4.2.2.20 of RFC 1122. Mallikarjun's proposed text revisions don't seem to capture the sense that this situation is a "fault" that SHOULD NOT be permitted to occur repeatedly or frequently. I agree that the current requirement "is severe", but OTOH implementations cannot rely on LLP retransmit or the use of "hidden" buffers of some form to recover from this entire class of situations. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp --=_alternative 00368EF188256D63_= Content-Type: text/html; charset="us-ascii"
David,
In your response that said: "I agree that the current requirement "is severe", but OTOH implementations cannot rely on LLP retransmit or the use of "hidden" buffers of some form to recover from this entire class of situations.".  This response caused me to pause and re-read it a number of times.  At the conclusion of the re-reading I am not sure that I agree or disagree.  This is because the missing buffer situation can be caused by things out of the control of the ULP.   And therefore not be a ULP error.

Suppose we have a application that manages many (perhaps thousands of connections) and therefore needs to manage a buffer pool and queue the approprate number of buffers (WQEs etc.) at various times.  There are clearly ways to determine, based on the link speed, the maximum number of buffers to queue, and at what interval to replenish the queue.  However, it is also possible that the OS many have some high priority system task which gets invoked just as the ULP is about to post additional buffers.  In this case it is possible that authorized responses from the various remote nodes  may use up the posted buffers, before the ULP can get control and post additional ones.

This is an important situation since even though the various remote nodes are given authorization (credits) to send in a message they may wait for some unknown time before sending the message and thus using the credit.   Since we can not determine when the message will be sent, the ULP will need to manage a buffer pool.  And the only thing it knows is the all the combined remote nodes can not exceed the bandwidth of the connection, and therefore upper buffer limit can be established.   When you take this upper limit and add some number of extra buffers, to handle the most probable cases of high priority system task getting in the way, you have a reasonable approach for managing a buffer pool.  However, some system vendors would like to protect customers from what ever impact  might be caused by the system itself or other higher priority tasks getting in the way of the ULP's buffer replenishment.  (Especially si! nce the connection, which may be struck by this problem is not predictable since it may be one of thousands that are sharing the buffer pool.)  Therefore, these systems vendors would like to pick a RNIC vendor that did not cause ULP connections to be ended because of some higher priority tasks getting in the way of the buffer replenishment.  I feel therefore, that having a HARD requirement that the connection MUST be terminated if a buffer is not available, is not the right thing to do.  

So I am not OK with treating a buffer shortage for receiving an "RDMA Send" messages as an error that must terminate the connection.  

 But I can not determine from your response whether you agree or not.  I am just not sure what words you are suggesting that we include.


.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: hufferd@us.ibm.com


Black_David@emc.com
Sent by: rddp-admin@ietf.org

07/14/2003 12:20 AM

       
        To:        cbm@rose.hp.com, rddp@ietf.org
        cc:        
        Subject:        RE: [rddp] Review comment on DDP untagged buffer model



On this topic, starting with Mallikarjun's original message:

> The DDP spec implies that the DDP Stream is to be
> dropped by implementations when there is no receive
> buffer to place an inbound untagged message.  This
> is mandating a specific type of implementation, and IMHO,
> is needless.  The DDP spec should instead allow for
> implementation flexibility in this area.  Those RNICs
> that can gracefully deal with lack of receive buffers
> via LLP means (such as TCP retransmit) should be
> allowed as well.  I can agree with the authors' original intent
> that ULP-level flow control and/or adequate buffer provisioning
> must be done to avoid needless retransmits, but mandating
> a connection drop on the first instance of hitting a (most
> likely a very transient) no-buffer window is severe.

plus one of Caitlin's comments that is very much on target:

> It make senese for the spec to allow the receiving stack some
> flexibility in handling a *fault* situation. Not having a buffer
> that should have been present for an authorized untagged message
> can indeed qualify as a fault.

I suggest that treating this sort of situation as a "fault" is a
good way to think about it - among the reasons for this is that
if TCP retransmit is the recovery mechanism, frequent and/or
routine usage of this sort of recovery has the potential to
run afoul of the "SHOULD be capable of queuing out-of-order
TCP segments" requirement in Section 4.2.2.20 of RFC 1122.

Mallikarjun's proposed text revisions don't seem to capture
the sense that this situation is a "fault" that SHOULD NOT be
permitted to occur repeatedly or frequently.  I agree that
the current requirement "is severe", but OTOH implementations
cannot rely on LLP retransmit or the use of "hidden" buffers
of some form to recover from this entire class of situations.

Thanks,
--David
----------------------------------------------------
David L. Black, Senior Technologist
EMC Corporation, 176 South St., Hopkinton, MA  01748
+1 (508) 293-7953             FAX: +1 (508) 293-7786
black_david@emc.com        Mobile: +1 (978) 394-7754
----------------------------------------------------




_______________________________________________
rddp mailing list
rddp@ietf.org
https://www1.ietf.org/mailman/listinfo/rddp

--=_alternative 00368EF188256D63_=-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 06:18:28 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA05207 for ; Mon, 14 Jul 2003 06:18:28 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c0PB-00005K-SP for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 06:18:02 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6EAI1cb000320 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 06:18:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c0PB-000055-PD for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 06:18:01 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA05192 for ; Mon, 14 Jul 2003 06:17:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c0P7-0003kj-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 06:17:57 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19c0P7-0003kg-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 06:17:57 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c0PA-0008WQ-Eq; Mon, 14 Jul 2003 06:18:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c0P8-0008W5-FY for rddp@optimus.ietf.org; Mon, 14 Jul 2003 06:17:58 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA05185 for ; Mon, 14 Jul 2003 06:17:54 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c0P4-0003kY-00 for rddp@ietf.org; Mon, 14 Jul 2003 06:17:54 -0400 Received: from maho3msx2.corp.emc.com ([128.221.11.32]) by ietf-mx with esmtp (Exim 4.12) id 19c0P3-0003kK-00 for rddp@ietf.org; Mon, 14 Jul 2003 06:17:54 -0400 Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <3W9QJQ4L>; Mon, 14 Jul 2003 06:17:21 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0C246717@corpmx14.us.dg.com> To: hufferd@us.ibm.com Cc: rddp@ietf.org Subject: RE: [rddp] Review comment on DDP untagged buffer model Date: Mon, 14 Jul 2003 06:07:30 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , John, > In your response that said: "I agree that the current requirement > "is severe", but OTOH implementations cannot rely on LLP retransmit > or the use of "hidden" buffers of some form to recover from this > entire class of situations.". This response caused me to pause and > re-read it a number of times. At the conclusion of the re-reading > I am not sure that I agree or disagree. This is because the missing > buffer situation can be caused by things out of the control of the > ULP. And therefore not be a ULP error. [ ... long explanation of statistical provisioning and OS interference in buffer pool replenishing snipped ... ] I think I'm in basic agreement. I believe that it is acceptable to use LLP recovery techniques for ULP-level problems such as not enough buffers due to a transient OS condition (a ULP-level problem, but not a ULP-level error) as long as such recovery occurs very infrequently. There are a number of reasons for this, starting with the performance and delivery consequences of an unnecessary retransmit. While I don't have any specific wording to propose, I think some well-placed SHOULDs/SHOULD NOTs plus explanation along the line that this sort of situation is a "fault" for which reliance on LLP retransmit or the like may be made in extremis (and infrequently) should suffice. I think the performance consequences will encourage a robustly-designed implementation to do the "right thing" here, but some direction needs to be given to implementers to tell them what that "right thing" is - when over-committing local resources, make sure that the situations in which otherwise legitimate traffic has to be dropped due to unavailable resources are very rare (SHOULD NOT happen, including the part of the "SHOULD NOT" about understanding the consequences). Thanks, --David -----Original Message----- From: John Hufferd [mailto:hufferd@us.ibm.com] Sent: Monday, July 14, 2003 5:56 AM To: Black_David@emc.com Cc: cbm@rose.hp.com; rddp@ietf.org; rddp-admin@ietf.org Subject: RE: [rddp] Review comment on DDP untagged buffer model David, In your response that said: "I agree that the current requirement "is severe", but OTOH implementations cannot rely on LLP retransmit or the use of "hidden" buffers of some form to recover from this entire class of situations.". This response caused me to pause and re-read it a number of times. At the conclusion of the re-reading I am not sure that I agree or disagree. This is because the missing buffer situation can be caused by things out of the control of the ULP. And therefore not be a ULP error. Suppose we have a application that manages many (perhaps thousands of connections) and therefore needs to manage a buffer pool and queue the approprate number of buffers (WQEs etc.) at various times. There are clearly ways to determine, based on the link speed, the maximum number of buffers to queue, and at what interval to replenish the queue. However, it is also possible that the OS many have some high priority system task which gets invoked just as the ULP is about to post additional buffers. In this case it is possible that authorized responses from the various remote nodes may use up the posted buffers, before the ULP can get control and post additional ones. This is an important situation since even though the various remote nodes are given authorization (credits) to send in a message they may wait for some unknown time before sending the message and thus using the credit. Since we can not determine when the message will be sent, the ULP will need to manage a buffer pool. And the only thing it knows is the all the combined remote nodes can not exceed the bandwidth of the connection, and therefore upper buffer limit can be established. When you take this upper limit and add some number of extra buffers, to handle the most probable cases of high priority system task getting in the way, you have a reasonable approach for managing a buffer pool. However, some system vendors would like to protect customers from what ever impact might be caused by the system itself or other higher priority tasks getting in the way of the ULP's buffer replenishment. (Especially since the connection, which may be struck by this problem is not predictable since it may be one of thousands that are sharing the buffer pool.) Therefore, these systems vendors would like to pick a RNIC vendor that did not cause ULP connections to be ended because of some higher priority tasks getting in the way of the buffer replenishment. I feel therefore, that having a HARD requirement that the connection MUST be terminated if a buffer is not available, is not the right thing to do. So I am not OK with treating a buffer shortage for receiving an "RDMA Send" messages as an error that must terminate the connection. But I can not determine from your response whether you agree or not. I am just not sure what words you are suggesting that we include. . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/System Group, San Jose CA Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Alt Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com Black_David@emc.com Sent by: rddp-admin@ietf.org 07/14/2003 12:20 AM To: cbm@rose.hp.com, rddp@ietf.org cc: Subject: RE: [rddp] Review comment on DDP untagged buffer model On this topic, starting with Mallikarjun's original message: > The DDP spec implies that the DDP Stream is to be > dropped by implementations when there is no receive > buffer to place an inbound untagged message. This > is mandating a specific type of implementation, and IMHO, > is needless. The DDP spec should instead allow for > implementation flexibility in this area. Those RNICs > that can gracefully deal with lack of receive buffers > via LLP means (such as TCP retransmit) should be > allowed as well. I can agree with the authors' original intent > that ULP-level flow control and/or adequate buffer provisioning > must be done to avoid needless retransmits, but mandating > a connection drop on the first instance of hitting a (most > likely a very transient) no-buffer window is severe. plus one of Caitlin's comments that is very much on target: > It make senese for the spec to allow the receiving stack some > flexibility in handling a *fault* situation. Not having a buffer > that should have been present for an authorized untagged message > can indeed qualify as a fault. I suggest that treating this sort of situation as a "fault" is a good way to think about it - among the reasons for this is that if TCP retransmit is the recovery mechanism, frequent and/or routine usage of this sort of recovery has the potential to run afoul of the "SHOULD be capable of queuing out-of-order TCP segments" requirement in Section 4.2.2.20 of RFC 1122. Mallikarjun's proposed text revisions don't seem to capture the sense that this situation is a "fault" that SHOULD NOT be permitted to occur repeatedly or frequently. I agree that the current requirement "is severe", but OTOH implementations cannot rely on LLP retransmit or the use of "hidden" buffers of some form to recover from this entire class of situations. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 08:47:32 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA16372 for ; Mon, 14 Jul 2003 08:47:32 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c2jP-0002vu-WF for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 08:47:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6ECl31X011270 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 08:47:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c2jP-0002vh-Rx for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 08:47:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA16303 for ; Mon, 14 Jul 2003 08:47:01 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c2jO-0006ia-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 08:47:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19c2jN-0006iX-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 08:47:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c2jN-0002vE-P3; Mon, 14 Jul 2003 08:47:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c2ir-0002u9-Af for rddp@optimus.ietf.org; Mon, 14 Jul 2003 08:46:29 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA16242 for ; Mon, 14 Jul 2003 08:46:26 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c2ip-0006hN-00 for rddp@ietf.org; Mon, 14 Jul 2003 08:46:27 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19c2ip-0006gL-00 for rddp@ietf.org; Mon, 14 Jul 2003 08:46:27 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 05B52244A64; Mon, 14 Jul 2003 08:46:00 -0400 (EDT) Date: Mon, 14 Jul 2003 07:45:52 -0500 From: Caitlin Bestler Subject: RE: [rddp] Review comment on DDP untagged buffer model To: John Hufferd , rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Monday, July 14, 2003, hufferd@us.ibm.com wrote: >Suppose we have a application that manages many (perhaps >thousands of connections) and therefore needs to manage a >buffer pool and queue the approprate number of buffers (WQEs >etc.) at various times. There are clearly ways to determine, >based on the link speed, the maximum number of buffers to >queue, and at what interval to replenish the queue. However, >it is also possible that the OS many have some high priority >system task which gets invoked just as the ULP is about to post >additional buffers. In this case it is possible that authorized >responses from the various remote nodes may use up the posted >buffers, before the ULP can get control and post additional >ones. Assuming that the ULP had a reasonable expectation that the posted buffers would become available, their failure to do so promptly would certainly qualify as a "fault" to me. On the other hand, an application that ran with only a few spare buffers, and counted on restocking them "just in time" without making any serious effort to deal with latency would be abusing the system. Having 100% predictable real-time behavior requires specialized execution environments. We certainly do not want to restrict RDMA deployment to such environments. But that does not mean the ULP can be sloppy and/or deliberately push the boundaries on providing an adequate buffer supply. In financing, it is acceptable to keep some of your funds in interest bearing accounts and only moving them into your checking account just-in-time. But it is not acceptable to routinely ignore all first billings and not even working on getting the funds to pay them when they are threatening to go to court. We need a similiar statement of expectations here. Prudent under- provisioning of buffers is simply wise resource management. When something goes wrong with it, it is acceptable to have a fault recovery strategy. What is not acceptable is to have any strategy that routinely *relies* upon retransmits, any more than it is acceptable to have a business plan that relies upon never paying an account until it is at least 90 days past due. Obviously, enforcement of such matters can never be better than SHOULD or SHOULD NOTs. But at least the intent needs to be clear. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 11:36:32 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA29223 for ; Mon, 14 Jul 2003 11:36:32 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c5My-0004xE-5N for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 11:36:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6EFa40q019043 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 11:36:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c5My-0004x4-2P for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 11:36:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA29218 for ; Mon, 14 Jul 2003 11:36:01 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c5Mx-0001jA-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 11:36:03 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19c5Mw-0001j7-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 11:36:02 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c5Mv-0004wi-Mh; Mon, 14 Jul 2003 11:36:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c5Ms-0004wL-QG for rddp@optimus.ietf.org; Mon, 14 Jul 2003 11:35:58 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA29215 for ; Mon, 14 Jul 2003 11:35:55 -0400 (EDT) From: pat_thaler@agilent.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c5Mr-0001j4-00 for rddp@ietf.org; Mon, 14 Jul 2003 11:35:57 -0400 Received: from msgbas1x.cos.agilent.com ([192.25.240.36]) by ietf-mx with esmtp (Exim 4.12) id 19c5Mq-0001j1-00 for rddp@ietf.org; Mon, 14 Jul 2003 11:35:57 -0400 Received: from relcos1.cos.agilent.com (relcos1.cos.agilent.com [130.29.152.239]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id 849F115675; Mon, 14 Jul 2003 09:35:53 -0600 (MDT) Received: from wcosbh22.cos.agilent.com (wcosbh22.cos.agilent.com [130.29.152.178]) by relcos1.cos.agilent.com (Postfix) with ESMTP id B4F7F1AD; Mon, 14 Jul 2003 09:35:55 -0600 (MDT) content-class: urn:content-classes:message Subject: RE: [rddp] Some Minimal end-to-end Buffer Rules Required Date: Mon, 14 Jul 2003 09:35:48 -0600 Message-ID: <1BEBA5E8600DD4119A50009027AF54A013856473@axcs04.cos.agilent.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thread-Topic: [rddp] Some Minimal end-to-end Buffer Rules Required X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 Thread-Index: AcNIn3JS1bSLoiivRFe79xI1k8dS9ABeabjQ To: , , Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable Caitlin, This proposed rule below is not practical as the DDP implementation has = no information on how many untagged messages the ULP has authorized. To = add that information would require constant communication between the = ULP and the DDP implementation to relay information about the credits = that had been passed to the peer. Such communication would be a = performance problem on the backplane. The ULP and not DDP should be = responsible for detecting that the peer has overrun its credits.=20 What is necessary to protect other connections in a shared buffer pool = situation is to deal with the problem of a peer that has tied up = excessive numbers of partially filled buffers. The ULP can only see = completions, so an attack is possible by sending segments for many = messages without completing any message. For instance, in the verbs = document, we put an alarm (with threshold settable by the ULP) for = excessive incomplete buffers in a receive queue. Regards, Pat -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Saturday, July 12, 2003 10:58 AM To: Stephen Bailey; rddp@ietf.org Subject: [rddp] Some Minimal end-to-end Buffer Rules Required I believe the following rules are required for untagged buffers -- Unauthorized untagged messages SHOULD terminate the DDP Stream. Failure to terminate an "over credit" DDP Stream MUST NOT imperil the availability of resources for any compliant DDP Stream. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 12:17:31 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02164 for ; Mon, 14 Jul 2003 12:17:31 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c60e-0007dL-HT for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 12:17:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6EGH4Ho029341 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 12:17:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c60e-0007dA-CH for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 12:17:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02108 for ; Mon, 14 Jul 2003 12:17:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c60c-0002QV-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 12:17:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19c60c-0002QQ-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 12:17:02 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c60a-0007ch-W5; Mon, 14 Jul 2003 12:17:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c60Y-0007cV-3P for rddp@optimus.ietf.org; Mon, 14 Jul 2003 12:16:58 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02103 for ; Mon, 14 Jul 2003 12:16:54 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c60W-0002QH-00 for rddp@ietf.org; Mon, 14 Jul 2003 12:16:56 -0400 Received: from emulex.emulex.com ([138.239.112.1]) by ietf-mx with esmtp (Exim 4.12) id 19c60V-0002Pb-00 for rddp@ietf.org; Mon, 14 Jul 2003 12:16:55 -0400 Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [138.239.73.11]) by emulex.emulex.com (8.12.9/8.12.8) with ESMTP id h6EGGILW017711 for ; Mon, 14 Jul 2003 09:16:19 -0700 (PDT) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id <3V4RZGLY>; Mon, 14 Jul 2003 12:16:18 -0400 Message-ID: <3356669BBE90C448AD4645C843E2BF28020C8261@xbl.ma.emulex.com> From: "Williams, Jim" To: rddp@ietf.org Subject: RE: [rddp] Review comment on DDP untagged buffer model Date: Mon, 14 Jul 2003 12:15:15 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , The standard use of TCP allows that the application (in this case RDDP) may stop accepting data from TCP at any point, and TCP will manage flow control, and buffer any interim data, until the application is able to accept more data. The definition of RDDP was intended to relieve TCP of this responsibility, and therefore the need to buffer a windows worth of data for an arbitrary period of time, by requiring RDDP is always able to accept data from TCP. So all received data will have a destination when it arrives, or if not, an error condition has occurred. So the question is we have relieved TCP from the responsibility of buffering data for application level flow control, but should we forbid it from doing so to help out RDDP? In the context of the above, Mallikarjun's proposal seems reasonable. I would raise one caution, though. My experience from VI is that failure to correctly handle flow control at the application level was one of the most common bugs and support issues we needed to deal with. The need to insure that receive buffers were always pre-posted before data was sent seemed to be a foreign concept to a lot of application programmers, and even those who fully understood it, often got it wrong. The technical arguments for doing receive buffer flow control at the ULP level rather than the RDMA level seem to have some merit, but the decision could come back to haunt us. In particular if a significant subset of RDMA implementations are able to handle flow control without help from the ULP, I would expect a proliferation of application code that requires this help in order to run correctly. The ultimate result would be a de facto requirement for RDMA to do flow control (or else forgo a large percentage of application code that requires it). So my order of preference might be something like the following. 1. Provide a clean architectural solution to allow RDMA to do receive buffer flow control. Applications would be allowed to send data before the receive buffers were available. RDMA layer would not allow data to be sent on the wire until receive buffers were posted and available. 2. Require ULP to do receive buffer flow control. In the event that ULP violates this requirement and sends data before receive buffer is available, the RDMA receiver MUST indicate a fatal error, and the ULP bug must be fixed. 3. Require ULP to do receive buffer flow control, but if it fails to do this properly, things will still work most of the time. Things may still work all the time with some RDMA implementations. Perhaps it's just a personal bias, but I hate things that work most of the time. I would much rather see things either work always, or fail quickly so they can be fixed. > -----Original Message----- > From: Mallikarjun C. [mailto:cbm@rose.hp.com] > Sent: Thursday, July 10, 2003 10:45 PM > To: rddp@ietf.org > Subject: [rddp] Review comment on DDP untagged buffer model > > > I have a comment on the DDP untagged buffer model > of the current DDP spec. > > The DDP spec implies that the DDP Stream is to be > dropped by implementations when there is no receive > buffer to place an inbound untagged message. This > is mandating a specific type of implementation, and IMHO, > is needless. The DDP spec should instead allow for > implementation flexibility in this area. Those RNICs > that can gracefully deal with lack of receive buffers > via LLP means (such as TCP retransmit) should be > allowed as well. I can agree with the authors' original intent > that ULP-level flow control and/or adequate buffer provisioning > must be done to avoid needless retransmits, but mandating > a connection drop on the first instance of hitting a (most > likely a very transient) no-buffer window is severe. > > I believe the DDP spec can allow this RNIC implementation > flexibility with minimal text changes. My review uncovered > only two areas of text that mandated the connection drop > requirement. > > section 3.2 (Protocol Overview), > " Note, however, that > the ULP needs to address flow control issues because if a DDP > Message arrives for an Untagged Buffer without an associated > receive ULP buffer, the DDP Message is dropped, the DDP Stream > is disabled for reception, and an error is reported to the ULP > at the Data Sink." > > and section 9.1 (Errors detected at the Data Sink). > " 2. The QN and MSN have an associated buffer that > allows Placement > of the payload." > > My suggestion is to change the text along the following lines: > Section 3.2 to: > " Note, however, that the ULP needs to address flow > control issues." > and section 9.1 to: > " 2. The QN and MSN have an associated buffer that allows > Placement of the payload. DDP implementations > MAY optionally consider lack of an associated buffer to be > not an error, but such implementations MUST ensure > that the DDP Segment is eventually recovered using LLP means > in a ULP-transparent way." > -- > Mallikarjun > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions > Hewlett-Packard MS 5668 > Roseville CA 95747 > cbm@rose.hp.com > > > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 12:46:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA03438 for ; Mon, 14 Jul 2003 12:46:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c6Si-0000b8-3K for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 12:46:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6EGk469002292 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 12:46:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c6Sh-0000at-SO for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 12:46:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA03407 for ; Mon, 14 Jul 2003 12:45:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c6Sg-0002jw-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 12:46:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19c6Sf-0002jt-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 12:46:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c6Sf-0000aN-VU; Mon, 14 Jul 2003 12:46:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c6SJ-0000Zl-I5 for rddp@optimus.ietf.org; Mon, 14 Jul 2003 12:45:39 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA03341 for ; Mon, 14 Jul 2003 12:45:35 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c6SH-0002iq-00 for rddp@ietf.org; Mon, 14 Jul 2003 12:45:37 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19c6SF-0002ii-00 for rddp@ietf.org; Mon, 14 Jul 2003 12:45:36 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id EA941244E33; Mon, 14 Jul 2003 12:44:45 -0400 (EDT) Date: Mon, 14 Jul 2003 11:44:38 -0500 From: Caitlin Bestler Subject: RE: [rddp] Some Minimal end-to-end Buffer Rules Required To: pat_thaler@agilent.com Cc: steph@cs.uchicago.edu, rddp@ietf.org X-Priority: 3 In-Reply-To: <1BEBA5E8600DD4119A50009027AF54A013856473@axcs04.cos.agilent.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Monday, July 14, 2003, pat_thaler@agilent.com wrote: >This proposed rule below is not practical as the DDP implementation has >no information on how many untagged messages the ULP has authorized. To >add that information would require constant communication between the >ULP and the DDP implementation to relay information about the credits >that had been passed to the peer. Such communication would be a >performance problem on the backplane. The ULP and not DDP should be >responsible for detecting that the peer has overrun its credits. > There are no more interactions than are required when fully provisioning the buffers. There is an inherent need to a) provide buffers and b) notify the ULP of completed reads. An example local interface would have the following capabilities: -- post buffer and credit to Endpoint Number of allowed untagged messages for Endpoint increments. Buffer is placed in receive queue for Endpoint (normally) or in an associated Shared Buffer Pool / Shared Receive Queue (if the Endpoint is so configured). -- grant credits to Endpoint without actually providing any buffers -- increasing the amount of underprovisioning. -- post buffer without credit to Endpoint To decrease the number of buffers. Only the first method/verb would be used during the processing of a typical transaction. A typical session would post some buffers, and if underprovisioning is to be used, some additional credits. As each completino is processed, a replacement buffer/credit is provided. Now the ULP is free to bundle these steps. If it wants to process three transactions and then post a single replacement with three buffers and credits. A more promising optimization might be to allow the replacement buffer and credit to be posted in the same operation as the reply message. I am not proposing a mandated procedure on *how* the ULP meters receives. Merely that it remain part of the loop. But given that the credit can easily be piggy-backed on re-enabling the buffer and/or sending the reply, I do not see how this can be a performance issue. >What is necessary to protect other connections in a shared buffer pool >situation is to deal with the problem of a peer that has tied up >excessive numbers of partially filled buffers. The ULP can only see >completions, so an attack is possible by sending segments for many >messages without completing any message. For instance, in the verbs >document, we put an alarm (with threshold settable by the ULP) for >excessive incomplete buffers in a receive queue. An Alarm that requires a ULP response is inherently suspect as a defense againt a Denial of Service attack. A static limit also does nothing to prevent an over-eager peer that always puts the next request on the wire too early. A static limit is fixed at N, it does not realize that it should be N-m because there are already m completed messages being processed. Virtually every ULP protocol seen to data limits the number of untagged messages that have not been responded to. Expanding the range of Valid MSN by posting a nrw receive operation (whether to the specific EP or to its pool) accomplishes this because it is done *after* the ULP hass finished processing a message. This sets the valid MSN window to the correct value, and can do so as part of inherent per-transaction interactions. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 13:01:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04728 for ; Mon, 14 Jul 2003 13:01:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c6hD-0001Xu-Ah for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 13:01:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6EH13BV005938 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 13:01:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c6hD-0001Xg-7T for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 13:01:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04671 for ; Mon, 14 Jul 2003 13:00:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c6hB-000337-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 13:01:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19c6hA-000334-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 13:01:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c6hB-0001XD-9Y; Mon, 14 Jul 2003 13:01:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c6h0-0001Wf-0a for rddp@optimus.ietf.org; Mon, 14 Jul 2003 13:00:50 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04655 for ; Mon, 14 Jul 2003 13:00:45 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c6gy-00032l-00 for rddp@ietf.org; Mon, 14 Jul 2003 13:00:48 -0400 Received: from atlrel8.hp.com ([156.153.255.206]) by ietf-mx with esmtp (Exim 4.12) id 19c6gw-00032d-00 for rddp@ietf.org; Mon, 14 Jul 2003 13:00:47 -0400 Received: from rosemail.rose.hp.com (rosemail.rose.hp.com [15.96.64.26]) by atlrel8.hp.com (Postfix) with ESMTP id 1840D1C0136D for ; Mon, 14 Jul 2003 13:00:45 -0400 (EDT) Received: from swathi (pal1nai166061.nsr.hp.com [15.244.166.61]) by rosemail.rose.hp.com with SMTP (8.7.1/8.7.3 SMKit7.02) id KAA19127 for ; Mon, 14 Jul 2003 10:00:42 -0700 (PDT) Message-ID: <008401c34a29$760e7780$1eeda051@rose.hp.com> From: "Mallikarjun C." To: Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required Date: Mon, 14 Jul 2003 09:58:44 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit I agree with Steph that the level of buffer management detail we're discussing doesn't exactly belong in an architecture doc. >The C notation is a terse way to express the model. >Don't read too much into it. Perhaps this is worth explicitly noting in the draft. I have issues with two of Caitlin's proposed "rules" wrt Untagged Messages. >-- Untagged messages may only be sent as explicitly allowed by > the Upper Layer Protocol. The ULP MUST define some mechanism > for granting permission to send Untagged messages. Let me note that "the ULP" in question here could actually be a protocol on top of what's traditionally called as "the ULP of the RDMAP". As an example, the flow control information and buffer size expectations could be set by iSCSI, even though it may not be directly running on top of RDMAP/DDP. >-- Unauthorized untagged messages SHOULD terminate the DDP Stream. - I saw this stated in an earlier email as well from you - how does the DDP layer determine an Untagged Message as authorized/unauthorized? - Note that this kind of wording again is leaning pretty close to the original text in that it essentially makes the same Stream termination recommendation in a strong way. As I stated at the beginning of this discussion, I prefer the DDP spec to *not* make any recommendations to the RNICs one way or the other. We should only have "SHOULD/SHOULD NOT"s around desirable ULP behavior (suggested text below). - I prefer some wording as David had implied on the original thread - "Any ULP that runs on DDP SHOULD consider the lack of a posted receive buffer to place an inbound Untagged Message of the ULP as an exception. Further, the ULP SHOULD NOT rely on the DDP layer gracefully handling this situation on a regular basis". --- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Hewlett-Packard MS 5668 Roseville CA 95747 cbm@rose.hp.com a.. To: Stephen Bailey , rddp@ietf.org b.. Subject: [rddp] Some Minimal end-to-end Buffer Rules Required c.. From: Caitlin Bestler d.. Date: Sat, 12 Jul 2003 12:58:26 -0500 e.. In-reply-to: f.. List-help: g.. List-id: IETF Remote Direct Data Placement (rddp) WG h.. List-post: i.. List-subscribe: , j.. List-unsubscribe: , k.. Sender: rddp-admin@ietf.org On Saturday, July 12, 2003, steph@cs.uchicago.edu wrote: >Caitlin, > >Sorry I've been slow responding to your previous message on this >topic. I'll roll it into this message. > >The executive summary is that you should be picking these bones with >the protocols, not the architecture. I think the first paragraph of >the architecture document makes this clear. > >> There are two distinct issues when dealing with the Untagged Message >> buffer model that must be dealt with seperately: flow control and >> provisioning of buffers. > >The architecture IS NOT defining an interface (API, or a looser Verbs >notion, or whatever else) for RDDP. It's only modelling what the >protocols do, in terms of messages and a set of endpoint behavior >constraints. The C notation is a terse way to express the model. >Don't read too much into it. > Perhaps it does not belong in the architecture draft, but that's where it was in the earlier drafts. The first draft required the local ULP to authorize untagged reception by posting a buffer. That was reduced to merely requiring the ULP to authorize untagged reception, but being silent about where buffers came from, and then to saying nothing about how untagged receptions are regulated. But the end-to-end protocol needs for certain rules about buffer usage to be explicitly stated. I suppose we can debate which draft they belong in precisely, but that does not mean these issues can be left totally to each implementation. There are certain required end-to-end behaviors needed to allow ULPs to work. I don't believe there is any debate on tagged buffers: -- Tagged messages may only access local memory to the extent that is explicitly authorized by the local ULP. -- The local interface MUST allow the local ULP to enable any contiguous subset of its address space through an STag. -- The local interface MAY allow other combinations of ULP accessible memory to be enabled for access through an STag. For example, zero-based buffers or maps formed from physical buffer lists. -- The local interface MUST NOT allow the ULP to enable an STag with memory access rights that the ULP does not itself already have. -- The remote peer MUST NOT use an advertised buffer in any way that is not explicitly authorized by the Upper Layer Protocol. -- The local ULP MUST use the local interface to enable STags for all access rights advertised to the remote peer. It MAY enable the STags for more rights, but SHOULD NOT do so if that places any other process or client at risk. -- The DDP Layer MUST terminate a DDP Stream if an incoming DDP Segment attempts to use an STag in a way that was not enabled by the local ULP. I believe the following rules are required for untagged buffers -- Untagged messages may only be sent as explicitly allowed by the Upper Layer Protocol. The ULP MUST define some mechanism for granting permission to send Untagged messages. -- Authorization for an untagged message MUST specify a maximum message size. Receipt of an untagged message that is larger MUST result in termination of the DDP Stream. -- Authorized untagged messages SHOULD be placed into a ULP accessible buffer. The local interface MUST allow the ULP to supply its own buffer for each authorized untagged message. The local interface MAY alternately allow the buffers to be filled from a shared resource. -- Unauthorized untagged messages SHOULD terminate the DDP Stream. Failure to terminate an "over credit" DDP Stream MUST NOT imperil the availability of resources for any compliant DDP Stream. -- The receiving side SHOULD have an adequate supply of buffers to handle all authorized untagged messages. However, this does not require pre-committment of one buffer for each authorized untagged message. Statistical sharing of buffer resources MAY be used. However the receiving side MUST NOT use this as a form of ex post facto flow control. With these rules, iWARP has extremely powerful and flexible flow control that is directly responsive to actual ULP needs. It is superior to the flow control available to non-RDMA IP ULPs. However, without these rules, an iWARP ULP is both more fragile and more of a security risk than a conventional non-RDMA ULP. There are several critical reasons for this: -- Transport layer buffers no longer pace the stream A principal benefit of RDMA protocols is eliminating dependence on transport layer buffering. But it is easy to overlook the fact that as hated as any bottleneck is, they also have side-effects that may have been beneficial. The side effect of being forced to pass-through the narrow capacity of transport layer buffering was some flow control. With an RDMA protocol there is no predicting where the next segment to arrive will be placed. Therefore it would be incorrect to hold it off because *one* of the logical destinations may have exhausted its capacity. The optimal interface between the LLP and DDP allows the LLP to pass each validated chunk/segment/whatever to the DDP as soon as it is validated and say, "here put this wherever it goes". Such an interface, if unchecked, would allow a remote peer to rapidly consume available buffers. -- The LLP cannot detect violations by the remote peer All DDP Segments look the same to the LLP. -- The ULP cannot detect violations by the remote peer Virtually all local interfaces that support RDMA protocols use a completion queue to deliver completions of untagged messages. Frequently these completion queues are: * Opaque. The ULP cannot knows the contents of the queue except by dequeuing them serially. * Shared across multiple DDP Streams. The net effect is that the ULP cannot know when an untagged message was received. By the time the completion is reaped, an indeterminant amount of time will have passed. The remote peer may have been granted permission during that time. Placing responsiblity for catching "cheaters" who sent one or two "extra" untagged messages early on the ULP will virtually guarantee that cheating will not be caught. If clients are competing for resources, for example when they are all using a common file server, it would be to the each of the clients advantage to cheat. Or at the minimum, because they are competing for resources, any client that has an "off by one" error will be far more motivated to fix it if they are sending one *too few* requests to the server. This could easily escalate into an arms race, where everyone assumes that the Server is counting on the clients cheating by one, and that playing by the rules would place the client at a competitive disadvantage versus other clients. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 13:21:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA06298 for ; Mon, 14 Jul 2003 13:21:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c70a-0002v8-20 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 13:21:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6EHL49k011220 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 13:21:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c70Z-0002ut-R2 for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 13:21:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA06279 for ; Mon, 14 Jul 2003 13:20:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c70X-0003SQ-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 13:21:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19c70X-0003SN-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 13:21:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c70X-0002uL-OM; Mon, 14 Jul 2003 13:21:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c70J-0002tv-5a for rddp@optimus.ietf.org; Mon, 14 Jul 2003 13:20:47 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA06273 for ; Mon, 14 Jul 2003 13:20:42 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c70H-0003S9-00 for rddp@ietf.org; Mon, 14 Jul 2003 13:20:45 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19c70G-0003Rl-00 for rddp@ietf.org; Mon, 14 Jul 2003 13:20:44 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 20D8C245777; Mon, 14 Jul 2003 13:20:18 -0400 (EDT) Date: Mon, 14 Jul 2003 12:20:11 -0500 From: Caitlin Bestler Subject: RE: [rddp] Review comment on DDP untagged buffer model To: "Williams, Jim" Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF28020C8261@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Monday, July 14, 2003, Jim.Williams@Emulex.com wrote: >So my order of preference might be something >like the following. > > 1. Provide a clean architectural solution to > allow RDMA to do receive buffer flow control. > Applications would be allowed to send data > before the receive buffers were available. > RDMA layer would not allow data to be sent > on the wire until receive buffers were posted > and available. > > 2. Require ULP to do receive buffer flow control. > In the event that ULP violates this requirement > and sends data before receive buffer is available, > the RDMA receiver MUST indicate a fatal error, > and the ULP bug must be fixed. > > 3. Require ULP to do receive buffer flow control, > but if it fails to do this properly, things will > still work most of the time. Things may still > work all the time with some RDMA implementations. > >Perhaps it's just a personal bias, but I hate things >that work most of the time. I would much rather see >things either work always, or fail quickly so they >can be fixed. I have an identical bias against things that work most of the time. I prefer my bugs to be blatant and painful. But, bugs and faults are inevitable. The question is how to deal with them. To my tastes, one of the least useful terms in use within software engineering is "fault tolerant". I am never tolerant of faults. I'll try to design my system to keep running in the face of faults, to contain any damage caused by a fault, to be robust in the face of non-compliance by the peer, etc. But it does not mean I am *tolerant* of those problems. So, with my draconian design-review hat on I can very simply state the local ULP MUST provide sufficient buffers to cover all authorized untagged messages that may arrive. Period. If you want to do statistical under-provisioning, fine, just be certain that you are providing enough buffering. But if you shift perspective from the design of a single ULP to the robustness of the entire system, different concerns leap to the forefront. Taking a zero tolerance approach has the danger of creating a cascading fault. Consider a ULP that was responsibly about to post extra buffers to a buffer pool, but delivery of the message has been delayed because of a system response to another unrelated fault. Perhaps a disk drive is down and the extra traffic to recreate the spare has clogged the IPC mechanisms. Or perhaps the portion of the ULP responsible is just being delayed in being swapped in. I prefer designing high volume servers as embedded systems with deterministic response times. But I would not limit RDMA servers to such environments. And if you require a buffer pool to be adequate under *all* circumstances, even in the face of other unrelated system faults, then you are essentially saying you cannot use buffer under-provisioning on a Unix or Windows server unless you are the *only* daemon running on it. An implementation preventing cascading of faults strikes me as not only permissible, but even commendable. An implementation that covers up sloppy ULP practices may be hard to forbid, but it certainly not commendable. It undermines interoperability. If you want to help the ULP, provide great diagnostics, not coddling. If the implementatino is covering up ULP flaws and not even logging a complaint then, in my opinion, it is going to far. Perhaps that is the real criteria in the back of my mind, its ok to fix a fault as long as it is infrequent enough that the log entries reporting it don't become an issue. If the user needs to suppress the log entries, then something has gone wrong. But back on your first point. There is no mechanism that will allow the sending side to defer sending a message. That would require an RDMA layer ack. I believe the decision not to piggy- back send-credits on untagged messages was the correct one. If the receiving side explicitly knows how many untagged messages the ULP is expecting, then there is a natural enforcemnt of flow control at the receiving side. If 'n' have been enabled, then that is the range of valid MSNs. An MSN out of that range is an invalid message. Period. It is a ULP fault at one end or the other. Tearing down the DDP Stream is the correct response. Valid MSN without a buffer can be a different situation, though. Unless we can determine that the absence of a buffer is *always* the fault of the receiving ULP I think we need to allow the stack to be robust. But the rule for ULP developers should remain: -- enable the next receive (typically by posting a buffer) -- THEN send the response. My experience has also been that lots of app programmers keep wanting to reverse those two. You can find rants about the same essential bug in Jackson System Design back in the 70s. The fact that this is a persistent application design mistake is exactly why we need to be very clear that the ULP *is* responsible for providing buffers *before* the messages arrive. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 13:46:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA07969 for ; Mon, 14 Jul 2003 13:46:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c7Oj-0004kL-QI for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 13:46:01 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6EHk1H7018241 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 13:46:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c7Oj-0004k8-NK for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 13:46:01 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA07942 for ; Mon, 14 Jul 2003 13:45:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c7Oh-0003sx-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 13:45:59 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19c7Og-0003su-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 13:45:58 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c7Oi-0004jg-DV; Mon, 14 Jul 2003 13:46:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c7OI-0004iy-Fq for rddp@optimus.ietf.org; Mon, 14 Jul 2003 13:45:34 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA07924 for ; Mon, 14 Jul 2003 13:45:31 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c7OG-0003sO-00 for rddp@ietf.org; Mon, 14 Jul 2003 13:45:32 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19c7OF-0003rw-00 for rddp@ietf.org; Mon, 14 Jul 2003 13:45:31 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id E6C8D2450D9; Mon, 14 Jul 2003 13:45:05 -0400 (EDT) Date: Mon, 14 Jul 2003 12:44:58 -0500 From: Caitlin Bestler Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required To: "Mallikarjun C." Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <008401c34a29$760e7780$1eeda051@rose.hp.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Monday, July 14, 2003, cbm@rose.hp.com wrote: >>-- Unauthorized untagged messages SHOULD terminate the DDP Stream. > >- I saw this stated in an earlier email as well from you - how does the > DDP layer determine an Untagged Message as authorized/unauthorized? > With a simple non-sharing Endpoint it is done by posting receive buffers. If the ULP has posted 7 receive buffers, it has enabled a range of 7 MSNs. If an incoming MSN is outside of that range it is invalid. This is very close to IB RC, the main difference is that there is no RDMAP layer message from the Data Sink to the Data Source that will cause the Data Source to hold the 8th message. iWARP is relying upon the ULP to enable the MSN *before* it enables its peer to send that message. This is a reasonable expectation. The ULP can enable new MSNs when it restocks the buffer pool or when it sends the reply message to its peer. No additional interactions are needed, but in any event, it is a trivial interaction compared to the other inteeractions already required on a per-transaction basis. >- Note that this kind of wording again is leaning pretty close > to the original text in that it essentially makes the same > Stream termination recommendation in a strong way. As I > stated at the beginning of this discussion, I prefer the DDP > spec to *not* make any recommendations to the RNICs one way or > the other. We should only have "SHOULD/SHOULD NOT"s around > desirable ULP behavior (suggested text below). > There is a very big difference. An Unauthorized message is one that the peer had no business sending. It indicates a software fault by the peer, or perhaps just in communication with the peer. The lack of a a buffer for an authorized message indicates a local fault in the stocking of the buffer pool. With some of the scenarios raised, this shortfall could easily be the result of system thrashing that was not the fault of local ULP. Requiring the ULP to take responsiblity for the worst case behavior of the OS and other applications is an unrealistic requirement for a daemon running in a multiple-application Unix or Windows environment. >- I prefer some wording as David had implied on the original thread - > "Any ULP that runs on DDP SHOULD consider the lack of a posted > receive buffer to place an inbound Untagged Message of the ULP > as an exception. Further, the ULP SHOULD NOT rely on the DDP > layer gracefully handling this situation on a regular basis". Again, if there is no buffer sharing, then the message is simply unauthorized. The remote peer is misbehaving, and could even be launching an attack. There is no reason to maintain the stream. Consider an Accounts Payable example: An incoming invoice is valid or invalid, no matter how much cash the company has on hand to pay it. Even if there is cash, Accounts Payable should not write a check that has not been authorized by some Department (the ULP in this case). But let us suppose the invoice is valid, and that the Department/ULP has authorized it knowing that it was allowed to do so under its budget. For reasons that are fundamentally identical, most companies do not segregate funds of a per department basis, but instead works from a shared pool of funds. So when a valid invoice arrives, but there is not enough cash to issue the check, what do you want Accounts Payable to do? The current RDDP draft would require the equivalent of treating it as an invalid invoice. That is probably not the response desired. Getting the funds transferred and *then* issuing the check is. Pretending not to have heard the DDP Segment is the equivalent of "sitting on" the invoice. But the same distinction applies: if the message is not authorized then it should not be placed, even if there is somewhere to place it. That is the equivalent of allowing Department X to overspend its budget and then not having enough cash to pay for something Department Y ordered that was compliant with its budget. (That's the part in the rules I proposed that says that accepting an unauthorized message MUST NOT adversely effect any other process or DDP Stream). When it is authorized, you SHOULD have somewhere to put it. Accounts Payable SHOULD have enough cash on hand to pay authorized invoices. When it doesn't, there has been a fault of some variety. Minimizing the damage of such a fault should be encouraged, as long as it does not imply acceptance of the fault-producing behavior. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 14:36:00 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA12471 for ; Mon, 14 Jul 2003 14:35:59 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c8Ae-0000DZ-Mq for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 14:35:32 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6EIZW9L000837 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 14:35:32 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c8Ae-0000DQ-JZ for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 14:35:32 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA12448 for ; Mon, 14 Jul 2003 14:35:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c8A9-0000BP-0Y; Mon, 14 Jul 2003 14:35:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c89S-0000Aj-Jm for rddp@optimus.ietf.org; Mon, 14 Jul 2003 14:34:18 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA12295 for ; Mon, 14 Jul 2003 14:34:14 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c89P-0004y3-00 for rddp@ietf.org; Mon, 14 Jul 2003 14:34:15 -0400 Received: from kairos.dsa.uqam.ca ([132.208.43.57]) by ietf-mx with esmtp (Exim 4.12) id 19c89O-0004xw-00 for rddp@ietf.org; Mon, 14 Jul 2003 14:34:14 -0400 Received: from localhost (bill@localhost) by Kairos.dsa.uqam.ca (8.11.4/8.11.4) with ESMTP id h6EHnVK05501; Mon, 14 Jul 2003 13:49:31 -0400 Date: Mon, 14 Jul 2003 13:49:31 -0400 (EDT) From: Bill Strahm X-X-Sender: To: Caitlin Bestler cc: "Mallikarjun C." , Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , I don't know how you can even SAY MUST drop the connection. Where a server may be running a hardware RNIC, a client very well may be running a software implementation over a normal L2 NIC. In this case, an Ethernet frame is received, there are no buffers posted to the NIC, so the frame is dropped on the floor. The OS may not even be aware that this has happened. Now if you want to remove the ability to run various RDMA protocols over normal L2 interfaces, requiring a Hardware approach, that is fine. I haven't seen this requirement in any of the documentation. Bill On Mon, 14 Jul 2003, Caitlin Bestler wrote: > On Monday, July 14, 2003, cbm@rose.hp.com wrote: > > >>-- Unauthorized untagged messages SHOULD terminate the DDP Stream. > > > >- I saw this stated in an earlier email as well from you - how does the > > DDP layer determine an Untagged Message as authorized/unauthorized? > > > With a simple non-sharing Endpoint it is done by posting receive > buffers. If the ULP has posted 7 receive buffers, it has enabled > a range of 7 MSNs. > > If an incoming MSN is outside of that range it is invalid. > > This is very close to IB RC, the main difference is that there > is no RDMAP layer message from the Data Sink to the Data Source > that will cause the Data Source to hold the 8th message. iWARP > is relying upon the ULP to enable the MSN *before* it enables > its peer to send that message. > > This is a reasonable expectation. The ULP can enable new MSNs > when it restocks the buffer pool or when it sends the reply > message to its peer. No additional interactions are needed, > but in any event, it is a trivial interaction compared to the > other inteeractions already required on a per-transaction basis. > > > >- Note that this kind of wording again is leaning pretty close > > to the original text in that it essentially makes the same > > Stream termination recommendation in a strong way. As I > > stated at the beginning of this discussion, I prefer the DDP > > spec to *not* make any recommendations to the RNICs one way or > > the other. We should only have "SHOULD/SHOULD NOT"s around > > desirable ULP behavior (suggested text below). > > > > There is a very big difference. > > An Unauthorized message is one that the peer had no business sending. > It indicates a software fault by the peer, or perhaps just in > communication with the peer. > > The lack of a a buffer for an authorized message indicates a local > fault in the stocking of the buffer pool. With some of the scenarios > raised, this shortfall could easily be the result of system thrashing > that was not the fault of local ULP. Requiring the ULP to take > responsiblity for the worst case behavior of the OS and other > applications is an unrealistic requirement for a daemon running > in a multiple-application Unix or Windows environment. > > > >- I prefer some wording as David had implied on the original thread - > > "Any ULP that runs on DDP SHOULD consider the lack of a posted > > receive buffer to place an inbound Untagged Message of the ULP > > as an exception. Further, the ULP SHOULD NOT rely on the DDP > > layer gracefully handling this situation on a regular basis". > > Again, if there is no buffer sharing, then the message is simply > unauthorized. The remote peer is misbehaving, and could even be > launching an attack. There is no reason to maintain the stream. > > > Consider an Accounts Payable example: > > An incoming invoice is valid or invalid, no matter how much cash the > company has on hand to pay it. Even if there is cash, Accounts Payable > should not write a check that has not been authorized by some Department > (the ULP in this case). > > But let us suppose the invoice is valid, and that the Department/ULP > has authorized it knowing that it was allowed to do so under its budget. > For reasons that are fundamentally identical, most companies do not > segregate funds of a per department basis, but instead works from > a shared pool of funds. > > So when a valid invoice arrives, but there is not enough cash to issue > the check, what do you want Accounts Payable to do? The current RDDP > draft would require the equivalent of treating it as an invalid invoice. > That is probably not the response desired. Getting the funds transferred > and *then* issuing the check is. Pretending not to have heard the > DDP Segment is the equivalent of "sitting on" the invoice. > > But the same distinction applies: if the message is not authorized then > it should not be placed, even if there is somewhere to place it. That is > the equivalent of allowing Department X to overspend its budget and then > not having enough cash to pay for something Department Y ordered that > was compliant with its budget. (That's the part in the rules I proposed > that says that accepting an unauthorized message MUST NOT adversely > effect any other process or DDP Stream). > > When it is authorized, you SHOULD have somewhere to put it. Accounts > Payable SHOULD have enough cash on hand to pay authorized invoices. > When it doesn't, there has been a fault of some variety. Minimizing > the damage of such a fault should be encouraged, as long as it does > not imply acceptance of the fault-producing behavior. > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 14:57:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA14564 for ; Mon, 14 Jul 2003 14:57:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c8VT-0002Fq-GI for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 14:57:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6EIv3s1008661 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 14:57:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c8VT-0002Fc-DJ for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 14:57:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA14506 for ; Mon, 14 Jul 2003 14:56:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c8VQ-0005Wn-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 14:57:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19c8VQ-0005Wk-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 14:57:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c8VR-0002Eu-O8; Mon, 14 Jul 2003 14:57:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19c8V7-0002E2-RD for rddp@optimus.ietf.org; Mon, 14 Jul 2003 14:56:41 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA14468 for ; Mon, 14 Jul 2003 14:56:38 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19c8V4-0005W3-00 for rddp@ietf.org; Mon, 14 Jul 2003 14:56:38 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19c8V3-0005VG-00 for rddp@ietf.org; Mon, 14 Jul 2003 14:56:38 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 9442F2450FE; Mon, 14 Jul 2003 14:56:11 -0400 (EDT) Date: Mon, 14 Jul 2003 13:56:03 -0500 From: Caitlin Bestler Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required To: Bill Strahm Cc: "Mallikarjun C." , rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Monday, July 14, 2003, bill@strahm.net wrote: >I don't know how you can even SAY MUST drop the connection. > >Where a server may be running a hardware RNIC, a client very >well may be running a software implementation over a normal L2 >NIC. In this case, an Ethernet frame is received, there are no >buffers posted to the NIC, so the frame is dropped on the >floor. The OS may not even be aware that this has happened. > >Now if you want to remove the ability to run various RDMA >protocols over normal L2 interfaces, requiring a Hardware >approach, that is fine. I haven't seen this requirement in any >of the documentation. Obviously a DDP Segment can be dropped anywhere along the way and the receiving DDP stack will not know it. So, allowing the DDP receiver to drop a message because of insufficient buffering is nothing different than any of the intermediate nodes dropping it. But the DDP receiver is the first one that a) Is capable of knowing whether or not the message was authorized. The intermediate routers, switches and Ethernet NICs simply do not know what a DDP Segment is. They especially do not know an MSN is, and they do not know how many receives the destination ULP has authorized. b) Is responsible for accepting final delivery. Keep in mind that the DDP layer MUST reset the Stream if the LLP *delivers* a DDP Segment to it and the payload cannot be placed. Once an LLP ack has been generated there is no option for dropping. An optimized LLP / DDP interface (which might be real or just a fancy euphemism for rototilling) allows the option of detecting the lack of buffer during Placement, but that option is *only* open during placement. The key here is not the availability of a buffer -- but the validity of the received message. Stating that an unauthorized untagged message should result in the DDP Stream being terminated is ultimately no different than stating that an invalid STag should result in a terminatin. In both cases, the peer had no right to send the message. Period. The issue is how to recognize that having the DDP receiver drop a valid DDP Segment that it lacks buffers for can be done without encouraging the ULP to use this capabilities in abusive ways. But obviously we have to allow it. Getting back to a de facto layered model, *not* dropping the Ethernet Frame that has a valid but unpaceable untagged message will ultimately mean dropping some *unknown* Ethernet Frame or frames. What if the frames you didn't allow the Ethernet NIC to capture contained Tagged messages that could have been placed? But we need to distinquish between validation of DDP Segments and buffer shortages. And within buffer shortages between those that the ULP could not realistically have prevented, and those caused by the ULP not fulfilling its responsibilities. We do NOT want to encourage the ULP to be dependent on features that are only plausible in thoroughly rototilled implementations. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 18:48:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA03569 for ; Mon, 14 Jul 2003 18:48:28 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cC71-0000op-3u for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 18:48:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6EMm3ic003143 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 18:48:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cC71-0000oc-0s for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 18:48:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA03543 for ; Mon, 14 Jul 2003 18:47:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cC6x-0001o2-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 18:47:59 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19cC6x-0001nz-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 18:47:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cC6y-0000o8-Qv; Mon, 14 Jul 2003 18:48:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cC6Y-0000ng-He for rddp@optimus.ietf.org; Mon, 14 Jul 2003 18:47:34 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA03493 for ; Mon, 14 Jul 2003 18:47:29 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cC6V-0001n2-00 for rddp@ietf.org; Mon, 14 Jul 2003 18:47:31 -0400 Received: from palrel13.hp.com ([156.153.255.238]) by ietf-mx with esmtp (Exim 4.12) id 19cC6U-0001mY-00 for rddp@ietf.org; Mon, 14 Jul 2003 18:47:30 -0400 Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel13.hp.com (Postfix) with ESMTP id 132BC1C01265 for ; Mon, 14 Jul 2003 15:47:21 -0700 (PDT) Received: from mk731915.cup.hp.com ([15.244.171.48]) by esmail.cup.hp.com (8.11.1/8.8.6) with ESMTP id h6EMg4620881 for ; Mon, 14 Jul 2003 15:42:04 -0700 (PDT) Message-Id: <5.1.0.14.2.20030714152125.0e716fd0@15.13.130.73> X-Sender: krause@15.13.130.73 X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Mon, 14 Jul 2003 15:31:57 -0700 To: rddp@ietf.org From: Michael Krause Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required In-Reply-To: References: <008401c34a29$760e7780$1eeda051@rose.hp.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 12:44 PM 7/14/2003 -0500, Caitlin Bestler wrote: >On Monday, July 14, 2003, cbm@rose.hp.com wrote: > > >>-- Unauthorized untagged messages SHOULD terminate the DDP Stream. > > > >- I saw this stated in an earlier email as well from you - how does the > > DDP layer determine an Untagged Message as authorized/unauthorized? > > >With a simple non-sharing Endpoint it is done by posting receive >buffers. If the ULP has posted 7 receive buffers, it has enabled >a range of 7 MSNs. > >If an incoming MSN is outside of that range it is invalid. This is implied knowledge outside the scope of the RDDP specifications. >This is very close to IB RC, the main difference is that there >is no RDMAP layer message from the Data Sink to the Data Source >that will cause the Data Source to hold the 8th message. IB has a send credit algorithm built into its transport acknowledgement wire protocol which enables it to understand whether placement is possible in one sense before it gets to the DDP layer. In addition, due to IB requiring strong ordering at the transport layer, i.e. no out-of-order placement is allowed, the implementation is simplified in this regard. >iWARP is relying upon the ULP to enable the MSN *before* it enables its >peer to send that message. As we developed / implemented IB, we learned quite a bit from the usage models. Basically, majority of ULP always provide an advertisement method and keep track of the number of outstanding operations allowed at a given time. As such, it was possible to eliminate this from the iWARP wire protocol as well as enable a well-layered implementation. >This is a reasonable expectation. The ULP can enable new MSNs when it >restocks the buffer pool or when it sends the reply message to its peer. >No additional interactions are needed, but in any event, it is a trivial >interaction compared to the other inteeractions already required on a >per-transaction basis. > > > >- Note that this kind of wording again is leaning pretty close > > to the original text in that it essentially makes the same > > Stream termination recommendation in a strong way. As I > > stated at the beginning of this discussion, I prefer the DDP > > spec to *not* make any recommendations to the RNICs one way or > > the other. We should only have "SHOULD/SHOULD NOT"s around > > desirable ULP behavior (suggested text below). > > > >There is a very big difference. > >An Unauthorized message is one that the peer had no business sending. It >indicates a software fault by the peer, or perhaps just in communication >with the peer. It indicates some form of fault / misunderstanding at either end of the communication as both sides can be at fault. >The lack of a a buffer for an authorized message indicates a local fault >in the stocking of the buffer pool. With some of the scenarios raised, >this shortfall could easily be the result of system thrashing >that was not the fault of local ULP. Requiring the ULP to take >responsiblity for the worst case behavior of the OS and other applications >is an unrealistic requirement for a daemon running in a >multiple-application Unix or Windows environment. Requiring the ULP to post buffers prior to advertising they can be targeted is not unreasonable if the resources are not shared. If the resources are shared, then the operating environment should be reasonably responsive to limit the number of shortages without requiring a rework of the ULP or requiring a stream to be shutdown. > >- I prefer some wording as David had implied on the original thread - > > "Any ULP that runs on DDP SHOULD consider the lack of a posted > > receive buffer to place an inbound Untagged Message of the ULP > > as an exception. Further, the ULP SHOULD NOT rely on the DDP > > layer gracefully handling this situation on a regular basis". > >Again, if there is no buffer sharing, then the message is simply >unauthorized. The remote peer is misbehaving, and could even be >launching an attack. There is no reason to maintain the stream. I prefer to leave this to an implementation to decide. I can envision implementations where the timing may just be off for posting fresh buffers and the peers have an implied as opposed to explicit understanding of the ULP credits. Mike _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 19:05:28 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA04246 for ; Mon, 14 Jul 2003 19:05:28 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cCNT-0002GG-78 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 19:05:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6EN53bc008655 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 19:05:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cCNT-0002FB-3l for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 19:05:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA04239 for ; Mon, 14 Jul 2003 19:04:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cCNP-0001zH-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 19:04:59 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19cCNP-0001zE-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 19:04:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cCNR-0002Eh-EM; Mon, 14 Jul 2003 19:05:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cCN4-0002E9-0Q for rddp@optimus.ietf.org; Mon, 14 Jul 2003 19:04:38 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA04231 for ; Mon, 14 Jul 2003 19:04:32 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cCN0-0001z1-00 for rddp@ietf.org; Mon, 14 Jul 2003 19:04:34 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19cCN0-0001yx-00 for rddp@ietf.org; Mon, 14 Jul 2003 19:04:34 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 41DE9244A0C; Mon, 14 Jul 2003 19:04:09 -0400 (EDT) Date: Mon, 14 Jul 2003 18:04:01 -0500 From: Caitlin Bestler Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required To: Michael Krause Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <5.1.0.14.2.20030714152125.0e716fd0@15.13.130.73> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Monday, July 14, 2003, krause@cup.hp.com wrote: >Requiring the ULP to post buffers prior to advertising they can >be targeted is not unreasonable if the resources are not >shared. If the resources are shared, then the operating >environment should be reasonably responsive to limit the number >of shortages without requiring a rework of the ULP or requiring >a stream to be shutdown. The exact same mechanism can be used when resources are shared. You post the replacement buffer to the Endpoint/QP. A new MSN is enabled. If buffers are shared, the buffer is placed in the shared pool. If buffers are not shared, it is reserved for that specific MSN (shared buffers are unordered, while stream specific buffers are ordered). Doing it this way: - preserves semantics between shared and unshared. If a message is invalid when your peer is fully provisioned, why should it become valid because your peer has under-provisioned? - allows ULPs to manage untagged credits independent of whether buffers pooling is in use or not. - ensure that there is always a simple test on the validity of the MSN. The security draft identifies flaws with this sort of shared resource plan. My objection remains that these are not flaws of the protocol. Knowing how many messages are enabled at the receiving QP has clearly been part of the envisioned architecture all along. The architecture draft went to last call with exactly that provision until I objected that it was unreasonable to require that a *buffer* be provided. If you review that discussion, neither I or anyone else objected to requiring the ULP to let the local endpoint/QP know how many MSNs it was enabling. Resource sharing can be achieved without giving up MSN bounds checking. MSN bounds checking is an important protection against resources being drained by over-eager clients. The only thing required is to maintain the "enabled inbound untagged message" count/credit for each stream -- the same one that is required for non-sharing endpoints anyway. A simple interface, more consistent behavior for the ULP designers to deal with (and someone designing a protocol does not want to deal with whether a server is using buffer sharing or not), a simpler implementation, and security holes plugged. What are you objecting to? _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 14 19:28:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA05308 for ; Mon, 14 Jul 2003 19:28:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cCjj-0003lh-LF for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 19:28:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6ENS305014479 for rddp-archive@odin.ietf.org; Mon, 14 Jul 2003 19:28:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cCjj-0003lS-Ht for rddp-web-archive@optimus.ietf.org; Mon, 14 Jul 2003 19:28:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA05243 for ; Mon, 14 Jul 2003 19:27:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cCjh-0002Di-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 19:28:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19cCjh-0002Df-00 for rddp-web-archive@ietf.org; Mon, 14 Jul 2003 19:28:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cCjh-0003kx-EB; Mon, 14 Jul 2003 19:28:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cCjO-0003bZ-FL for rddp@optimus.ietf.org; Mon, 14 Jul 2003 19:27:42 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA05204 for ; Mon, 14 Jul 2003 19:27:38 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cCjM-0002D1-00 for rddp@ietf.org; Mon, 14 Jul 2003 19:27:40 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19cCjM-0002CD-00 for rddp@ietf.org; Mon, 14 Jul 2003 19:27:40 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 78C44244C0C; Mon, 14 Jul 2003 19:27:14 -0400 (EDT) Date: Mon, 14 Jul 2003 18:27:06 -0500 From: Caitlin Bestler Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required To: Michael Krause Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <5.1.0.14.2.20030714152125.0e716fd0@15.13.130.73> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Monday, July 14, 2003, krause@cup.hp.com wrote: >>With a simple non-sharing Endpoint it is done by posting receive >>buffers. If the ULP has posted 7 receive buffers, it has enabled >>a range of 7 MSNs. >> >>If an incoming MSN is outside of that range it is invalid. > >This is implied knowledge outside the scope of the RDDP specifications. Section 9.1 of the DDP spec, point 5 (page 27): > For non-zero length Untagged DDP Segments, the DDP Segment MUST be > validated before Placement by verifying: > ... > > 5. For DDP Messages using Untagged Buffer model, the Message > Sequence Number falls in the range of legal Message Sequence > Numbers, for the queue defined by the QN. The legal range is > defined as being between the MSN value assigned to the first > available buffer for a specific QN and the MSN value assigned to > the last available buffer for a specific QN. >From Hemal Shah's presentation at the November 2002 IETF: >Untagged Buffer data transfer model >* Enables data transfer without requiring buffer advertisement >* Receiver can queue up a series of ULP buffers to specify the > order in which the buffers will be consumed (receiver-based > ULP buffer management) >* Each Untagged DDP Message from the Data Source consumes an > Untagged Buffer at the Data Sink >* Requires associating a receive ULP buffer (based on MSN and > QN fields) for each DDP Message If an Untagged DDP Message > arrives without an associated Untagged Buffer, the DDP Message > is dropped and DDP Stream is terminated It strikes me that the concept that the valid range of MSNs is "unknown" is a revisionist interpretation. Validation of MSNs was part of the protocol before resource pooling was proposed, and resource pooling can be achieved without elimination of MSN validation. Eliminating MSN validation makes implementation independent end-to-end ULP flow control more difficult to enforce. It makes implementations more complex (because the range of valid MSNs is no longer determined *before* a packet arrives), and it creates security concerns. The only change that was proposed was eliminating the need to specify a specific buffer when enabling an untagged receive. I maintain that this is the *only* change required. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 15 12:10:44 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA25698 for ; Tue, 15 Jul 2003 12:10:44 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cSNd-00065j-Nk for rddp-archive@odin.ietf.org; Tue, 15 Jul 2003 12:10:19 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6FGAHDe023416 for rddp-archive@odin.ietf.org; Tue, 15 Jul 2003 12:10:17 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cSNd-00065b-JJ for rddp-web-archive@optimus.ietf.org; Tue, 15 Jul 2003 12:10:17 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA25687 for ; Tue, 15 Jul 2003 12:10:12 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cSNc-0003Jw-00 for rddp-web-archive@ietf.org; Tue, 15 Jul 2003 12:10:16 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19cSNW-0003Jp-00 for rddp-web-archive@ietf.org; Tue, 15 Jul 2003 12:10:10 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cSNN-00063l-A7; Tue, 15 Jul 2003 12:10:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cSNJ-00063O-Gj for rddp@optimus.ietf.org; Tue, 15 Jul 2003 12:09:57 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA25676 for ; Tue, 15 Jul 2003 12:09:52 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cSNI-0003Jh-00 for rddp@ietf.org; Tue, 15 Jul 2003 12:09:56 -0400 Received: from e35.co.us.ibm.com ([32.97.110.133]) by ietf-mx with esmtp (Exim 4.12) id 19cSMx-0003Ia-00 for rddp@ietf.org; Tue, 15 Jul 2003 12:09:35 -0400 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e35.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h6FG7oc8270382 for ; Tue, 15 Jul 2003 12:07:50 -0400 Received: from d03nm115.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay02.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h6FG7jw5181366 for ; Tue, 15 Jul 2003 10:07:46 -0600 To: rddp@ietf.org MIME-Version: 1.0 Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 From: John Hufferd Message-ID: Date: Tue, 15 Jul 2003 09:07:48 -0700 X-MIMETrack: Serialize by Router on D03NM115/03/M/IBM(Release 6.0.1 [IBM]|June 10, 2003) at 07/15/2003 10:07:50, Serialize complete at 07/15/2003 10:07:50 Content-Type: multipart/alternative; boundary="=_alternative 0058952F88256D64_=" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multipart message in MIME format. --=_alternative 0058952F88256D64_= Content-Type: text/plain; charset="us-ascii" I believe we are moving towards general agreement on not being hard on the ULP that uses Buffers in a buffer pool or Shared RQ type of an operation (in other words, we seem to be agreeing that we may find words to permit special handling on a, hopefully rare missing buffer/WQE event.) So with that assumption, I think we might still need to be careful as we talk through the words that imply a connection must be drop for non shared environments when a buffer (or WQE) is missing. As a point of discussion let me give you an example of an application which in normal situations always have its buffers posted for the remote node to use when issuing RDMA "Send" type messages, but can have problems in special cases. An example of the special case may be an Error/Exception handling async type message. Generally these types of messages are completely unpredictable about when they may occur, and they also may have an unpredictable number of such message. (Many times when things go wrong, lots of things also seem to go wrong.) Though this is a very rare situation, the important point is that often the timing and number of messages are not well predictable. Also, it is not reasonable for the failing node to negotiate for more credits as it is attempting to send messages about its current trauma. So the point I want to make that many times it is important for there to be a graceful handling of a missing buffer, even in the case of non shared buffers/queues. Based on this I am aware of system vendors that would like to build their systems from RNICs that would permit graceful handling of missing buffers even in the non shared case. And I think we should give careful consideration to this desire. . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/System Group, San Jose CA Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Alt Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com --=_alternative 0058952F88256D64_= Content-Type: text/html; charset="us-ascii"
I believe we are moving towards general agreement on not being hard on the ULP that uses Buffers in a buffer pool or Shared RQ type of an operation (in other words, we seem to be agreeing that we may find words to permit special handling on a, hopefully rare missing buffer/WQE event.)

So with that assumption,  I think we might still need to be careful as we talk through the words that imply a connection must be drop for non shared environments when a buffer (or WQE) is missing.

As a point of discussion let me give you an example of an application which in normal situations always have its buffers posted for the remote node to use when issuing RDMA "Send" type messages, but can have problems in special cases.  An example of the special case may be an Error/Exception handling async type message.  Generally these types of messages are completely unpredictable about when they may occur, and they also may have an unpredictable number of such message.  (Many times when things go wrong, lots of things also seem to go wrong.)

Though this is a very rare situation, the important point is that often the timing and number of messages are not well predictable.  Also, it is not reasonable for the failing node to negotiate for more credits as it is attempting to send messages about its current trauma.

So the point I want to make that many times it is important for there to be a graceful handling of a missing buffer, even in the case of non shared buffers/queues.  Based on this I am aware of system vendors that would like to build their systems from RNICs that would permit graceful handling of missing buffers even in the non shared case.  And I think we should give careful consideration to this desire.




.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: hufferd@us.ibm.com --=_alternative 0058952F88256D64_=-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 15 12:45:35 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA26545 for ; Tue, 15 Jul 2003 12:45:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cSvL-0007Ub-TZ for rddp-archive@odin.ietf.org; Tue, 15 Jul 2003 12:45:10 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6FGj7v9028802 for rddp-archive@odin.ietf.org; Tue, 15 Jul 2003 12:45:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cSvL-0007UT-ME for rddp-web-archive@optimus.ietf.org; Tue, 15 Jul 2003 12:45:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA26538 for ; Tue, 15 Jul 2003 12:45:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cSvK-0003eg-00 for rddp-web-archive@ietf.org; Tue, 15 Jul 2003 12:45:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19cSvE-0003ed-00 for rddp-web-archive@ietf.org; Tue, 15 Jul 2003 12:45:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cSvE-0007Sk-Uv; Tue, 15 Jul 2003 12:45:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cSv9-0007SE-JW for rddp@optimus.ietf.org; Tue, 15 Jul 2003 12:44:55 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA26531 for ; Tue, 15 Jul 2003 12:44:50 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cSv8-0003eS-00 for rddp@ietf.org; Tue, 15 Jul 2003 12:44:54 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19cSux-0003br-00 for rddp@ietf.org; Tue, 15 Jul 2003 12:44:43 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id C696D244D53; Tue, 15 Jul 2003 12:43:28 -0400 (EDT) Date: Tue, 15 Jul 2003 11:43:22 -0500 From: Caitlin Bestler Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required To: John Hufferd Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Tuesday, July 15, 2003, hufferd@us.ibm.com wrote: >I believe we are moving towards general agreement on not being >hard on the ULP that uses Buffers in a buffer pool or Shared RQ >type of an operation (in other words, we seem to be agreeing >that we may find words to permit special handling on a, >hopefully rare missing buffer/WQE event.) > >So with that assumption, I think we might still need to be >careful as we talk through the words that imply a connection >must be drop for non shared environments when a buffer (or WQE) >is missing. > Agreed. The distinction I prefer is that the credit MUST be there (i.e., the WQE) but that the buffer itself can be under-provisioned and therefore on rare occassions might be absent. The first is a violation by the peer, the latter is a local fault. Recovery from local faults are not only permissable, but commendable. >As a point of discussion let me give you an example of an >application which in normal situations always have its buffers >posted for the remote node to use when issuing RDMA "Send" type >messages, but can have problems in special cases. An example >of the special case may be an Error/Exception handling async >type message. Generally these types of messages are completely >unpredictable about when they may occur, and they also may have >an unpredictable number of such message. (Many times when >things go wrong, lots of things also seem to go wrong.) > The error/exception handling is a double-edged sword here. If the server under-provisions buffers, and there is a rare surge of outside traffice that depletes the buffer pool and triggers a "low buffer" exception. Under normal circumtances this exception would be processed in time for the buffer pool to be replenished. But because of another operation on the same OS being at its peak activity level the exception is not processed quickly enough. Recovering from such a fault is a very good idea. As I commeneted previously, it is unreasonable to expect a daemon on a Unix/Windows type system to take responsibility for worst case performance. You cannot build absolute guarantees when your OS only gives you statistical guarantees. But I believe that reliance on an Error/Exception handler to respond to what may be a Denial of Service attack is a broken interface with severe security implications. >Though this is a very rare situation, the important point is >that often the timing and number of messages are not well >predictable. Also, it is not reasonable for the failing node >to negotiate for more credits as it is attempting to send >messages about its current trauma. > If the ULP is responsible for pacing, then the number of untagged messages is not unpredictable. It is known. If it is known to the ULP it can easily be communicated to the DDP stack. This is traditionally done by posting receive operations. A minimal change is to simply allow posting of receives that specify a buffer pool rather than a buffer. Further optimizations can be made by piggy-backing this operation on sending the reply. >So the point I want to make that many times it is important for >there to be a graceful handling of a missing buffer, even in >the case of non shared buffers/queues. Based on this I am >aware of system vendors that would like to build their systems >from RNICs that would permit graceful handling of missing >buffers even in the non shared case. And I think we should >give careful consideration to this desire. > > My fear is similar to one that Jim Williams expressed. I do not want RNIC vendors taking on duties that belong to the ULP and presenting them as though they were RDMAP/DDP functions. If someone were to build a DAFS server on top of an RNIC that did not enforce untagged buffer limits then the DAFS server would have no reason to seriously negotiate the per-session in-flight limits. Such a DAFS server would fail to work on a compliant RNIC. Additionally, on a pure-network basis, ULP level flow control is vital to being network friendly. If the ULP *wants* to issue more credits than it advertises, then it should be allowed to do so. But having the iWARP stack do so for it be default is dangerous and should be avoided. Even so, I would not forbid it. The rule I proposed is that accepting an unauthorized untagged message MUST NOT have an adverse impact on the ability to accept an authorized message on any other stream. This is equivalent to a common rate- enforcement policy, that an over-rate cell/packet must never be forwarded in preference to a rate-compliant cell/packet. Arguments can be made either way as to when forwarding over-rate cells/packets that do not block legitimate traffic becomes a bad idea. Most will agree that there is a point that the sender is deliberately abusing the system, and the cell/packet will be dropped even if the network is otherwise completely idle. But that can vary, what's important is that a cheater must not be able to drain resources from a non-cheater. Accepting non-authorized untagged messages from a buffer pool or shared receive queue runs *exactly* that risk. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 15 12:54:37 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA26806 for ; Tue, 15 Jul 2003 12:54:37 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cT43-0007rh-Ov for rddp-archive@odin.ietf.org; Tue, 15 Jul 2003 12:54:09 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6FGs7dv030229 for rddp-archive@odin.ietf.org; Tue, 15 Jul 2003 12:54:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cT43-0007rU-M1 for rddp-web-archive@optimus.ietf.org; Tue, 15 Jul 2003 12:54:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA26786 for ; Tue, 15 Jul 2003 12:54:01 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cT41-0003iv-00 for rddp-web-archive@ietf.org; Tue, 15 Jul 2003 12:54:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19cT3w-0003is-00 for rddp-web-archive@ietf.org; Tue, 15 Jul 2003 12:54:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cT3x-0007oI-Dq; Tue, 15 Jul 2003 12:54:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cT3h-0007nz-Q0 for rddp@optimus.ietf.org; Tue, 15 Jul 2003 12:53:45 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA26779 for ; Tue, 15 Jul 2003 12:53:38 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cT3e-0003im-00 for rddp@ietf.org; Tue, 15 Jul 2003 12:53:42 -0400 Received: from atlrel9.hp.com ([156.153.255.214]) by ietf-mx with esmtp (Exim 4.12) id 19cT3T-0003id-00 for rddp@ietf.org; Tue, 15 Jul 2003 12:53:31 -0400 Received: from rosemail.rose.hp.com (rosemail.rose.hp.com [15.96.64.26]) by atlrel9.hp.com (Postfix) with ESMTP id 960621C012C9; Tue, 15 Jul 2003 12:53:08 -0400 (EDT) Received: from swathi (pal1nai161233.nsr.hp.com [15.244.161.233]) by rosemail.rose.hp.com with SMTP (8.7.1/8.7.3 SMKit7.02) id JAA09574; Tue, 15 Jul 2003 09:53:06 -0700 (PDT) Message-ID: <013301c34af1$90ad8310$1eeda051@rose.hp.com> From: "Mallikarjun C." To: , "John Hufferd" References: Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required Date: Tue, 15 Jul 2003 09:50:50 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit I agree with John that the core issue is the same regardless of S-RQ. In my opinion, the sharing/non-sharing of buffers is an orthogonal issue to graceful handling. While the scope of sharing (and whether to share) across remote peers is determined by a ULP based on trust/DoS considerations wrt the peers, the need for graceful handling is an inherent characteristic of the ULP/UULP's design. Even in the absence of any sharing, no ULP would like a severe stream termination approach on hitting this exception. As I said at the beginning of the untagged model discussion - > The Data Sink ULP would still be required to authorize each reception Generally, I agree. It may however be the case for some ULPs (such as iSCSI) that they may not be able to precisely predict every possible ULP PDU though the peers know about >95% of the PDUs with certainty (and can provision for them). The ULP in this case can post a few extra "slack" buffers, but it still is theoretically possible in some very corner cases that a local buffer pool may transiently run out. John Hufferd gave an example of processing delays that could also lead to this same scenario. John is elaborating this with iSCSI-specific examples (a bunch of immediate Abort Task requests is another example), and these should give an idea of the issue that some of us are seeing with the current mandate of the DDP spec. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Hewlett-Packard MS 5668 Roseville CA 95747 cbm@rose.hp.com ----- Original Message ----- From: "John Hufferd" To: Sent: Tuesday, July 15, 2003 9:07 AM Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required > I believe we are moving towards general agreement on not being hard on the > ULP that uses Buffers in a buffer pool or Shared RQ type of an operation > (in other words, we seem to be agreeing that we may find words to permit > special handling on a, hopefully rare missing buffer/WQE event.) > > So with that assumption, I think we might still need to be careful as we > talk through the words that imply a connection must be drop for non shared > environments when a buffer (or WQE) is missing. > > As a point of discussion let me give you an example of an application > which in normal situations always have its buffers posted for the remote > node to use when issuing RDMA "Send" type messages, but can have problems > in special cases. An example of the special case may be an > Error/Exception handling async type message. Generally these types of > messages are completely unpredictable about when they may occur, and they > also may have an unpredictable number of such message. (Many times when > things go wrong, lots of things also seem to go wrong.) > > Though this is a very rare situation, the important point is that often > the timing and number of messages are not well predictable. Also, it is > not reasonable for the failing node to negotiate for more credits as it is > attempting to send messages about its current trauma. > > So the point I want to make that many times it is important for there to > be a graceful handling of a missing buffer, even in the case of non shared > buffers/queues. Based on this I am aware of system vendors that would > like to build their systems from RNICs that would permit graceful handling > of missing buffers even in the non shared case. And I think we should > give careful consideration to this desire. > > > > > . > . > John L. Hufferd > Senior Technical Staff Member (STSM) > IBM/System Group, San Jose CA > Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 > Alt Office: (408) 997-6136, Cell: (408) 499-9702 > Internet Address: hufferd@us.ibm.com _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 15 13:49:34 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA27889 for ; Tue, 15 Jul 2003 13:49:33 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cTvH-0001ry-Kh for rddp-archive@odin.ietf.org; Tue, 15 Jul 2003 13:49:07 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6FHn7HC007180 for rddp-archive@odin.ietf.org; Tue, 15 Jul 2003 13:49:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cTvH-0001rj-Fl for rddp-web-archive@optimus.ietf.org; Tue, 15 Jul 2003 13:49:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA27885 for ; Tue, 15 Jul 2003 13:49:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cTvF-000489-00 for rddp-web-archive@ietf.org; Tue, 15 Jul 2003 13:49:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19cTv9-000486-00 for rddp-web-archive@ietf.org; Tue, 15 Jul 2003 13:48:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cTvA-0001q3-Gh; Tue, 15 Jul 2003 13:49:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cTux-0001pr-CI for rddp@optimus.ietf.org; Tue, 15 Jul 2003 13:48:47 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA27882 for ; Tue, 15 Jul 2003 13:48:42 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cTuv-000480-00 for rddp@ietf.org; Tue, 15 Jul 2003 13:48:45 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19cTuk-00047m-00 for rddp@ietf.org; Tue, 15 Jul 2003 13:48:34 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id F23CE24587C; Tue, 15 Jul 2003 13:47:46 -0400 (EDT) Date: Tue, 15 Jul 2003 12:47:39 -0500 From: Caitlin Bestler Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required To: "Mallikarjun C." Cc: rddp@ietf.org, John Hufferd X-Priority: 3 In-Reply-To: <013301c34af1$90ad8310$1eeda051@rose.hp.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Tuesday, July 15, 2003, cbm@rose.hp.com wrote: >I agree with John that the core issue is the same regardless of S-RQ. > Agreed. A "non-sharing" endpoint is simply sharing over a pool of one. So, regardless of how big the pool is, I believe there is full agreement that a) under-provisioning that pool is a valid local option, b) this should not be abused, that is you should not expect the pool to go dry and c) graceful recovery from miscalculations on the pool size are permissable and even commendable. The remaining issue is ULP flow control. Traditionally, RDMA protocols have enforced ULP flow control by requiring each untagged message (or Send) to be explicitly authorized at the receiving end. Buffer pooling / underprovisioning can be achieved without eliminating this fundamental end-to-end safety check. Further, because other modes of flow control are *less* effective once RDMA is enabled it is important that we leave ULP flow control fully intact. >In my opinion, the sharing/non-sharing of buffers is an >orthogonal issue to graceful handling. While the scope of >sharing (and whether to share) across remote peers is >determined by a ULP based on trust/DoS considerations wrt the >peers, the need for graceful handling is an inherent >characteristic of the ULP/UULP's design. Even in the absence >of any sharing, no ULP would like a severe stream termination >approach on hitting this exception. > Failure to bound the range of valid MSNs creates a penalty on implementations. Further, it is a type of slippery slope that should be avoided. If *some* RNICs allow servers to ignore their responsibility for end-to-end flow control then applications will be developed that fail to perform this role -- and it is the compliant NICs that will be blamed when those applications fail. We don't allow the RNIC to create a spare buffer for an invalid STag and keep the data somewhere until the application figures out where it was really meant to go. We expect the peer to follow the protocol and send the correct STag. I fail to see why we should allow the RNIC to accept an invalid MSN, consume buffers that might be needed for legitimate messages, and hold it until it is time to deliver it on the assumption that it was really a valid message. Accepting "out of window" TCP segments or SCTP data chunks is considered to be a security hole, not a feature. Why are invalid untagged messages so deserving of coddling? >As I said at the beginning of the untagged model discussion - > > > The Data Sink ULP would still be required to authorize > > each reception > > Generally, I agree. It may however be the case for some ULPs > (such as iSCSI) that they may not be able to precisely predict > every possible ULP PDU though the peers know about >95% of > the PDUs with certainty (and can provision for them). The ULP > in this case can post a few extra "slack" buffers, but it > still is theoretically possible in some very corner cases that > a local buffer pool may transiently run out. > If the ULP does not know how many untagged messages its peer is authorized to send then it is not doing flow-control. It is probably relying on LLP flow control to keep it out of trouble. That does not work with RDMA because it has been designed to keep the LLP from being a bottleneck. We don't want to add additional network messages to allow iWARP acks and credit transmissions, therefore we need to rely on the ULPs to send credits. That is not some vague philosophical assertion, it is an important design point that must be taken into account by anyone who designs an upper layer protocol to run over RDMAP/DDP. Negotiate a limit on "asynchronous" messages, and wait for a ULP ack before sending more. It's very simple, and very secure. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 15 18:59:32 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA10284 for ; Tue, 15 Jul 2003 18:59:32 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cYlI-0007Zi-La for rddp-archive@odin.ietf.org; Tue, 15 Jul 2003 18:59:09 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6FMx8YO029116 for rddp-archive@odin.ietf.org; Tue, 15 Jul 2003 18:59:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cYlI-0007ZX-BI for rddp-web-archive@optimus.ietf.org; Tue, 15 Jul 2003 18:59:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA10281 for ; Tue, 15 Jul 2003 18:59:01 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cYlF-0006t8-00 for rddp-web-archive@ietf.org; Tue, 15 Jul 2003 18:59:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19cYl9-0006t5-00 for rddp-web-archive@ietf.org; Tue, 15 Jul 2003 18:58:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cYlB-0007Z1-4B; Tue, 15 Jul 2003 18:59:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cYkJ-0007Yf-K1 for rddp@optimus.ietf.org; Tue, 15 Jul 2003 18:58:10 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA10231 for ; Tue, 15 Jul 2003 18:58:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cYkG-0006s8-00 for rddp@ietf.org; Tue, 15 Jul 2003 18:58:04 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19cYk5-0006rn-00 for rddp@ietf.org; Tue, 15 Jul 2003 18:57:53 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 2987924530C; Tue, 15 Jul 2003 18:56:59 -0400 (EDT) Date: Tue, 15 Jul 2003 17:56:50 -0500 From: Caitlin Bestler Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required To: "Mallikarjun C." Cc: rddp@ietf.org, John Hufferd X-Priority: 3 In-Reply-To: <013301c34af1$90ad8310$1eeda051@rose.hp.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Tuesday, July 15, 2003, cbm@rose.hp.com wrote: > Generally, I agree. It may however be the case for some ULPs > (such as iSCSI) that they may not be able to precisely predict > every possible ULP PDU though the peers know about >95% of > the PDUs with certainty (and can provision for them). The ULP > in this case can post a few extra "slack" buffers, but it > still is theoretically possible in some very corner cases that > a local buffer pool may transiently run out. Some further thoughts on the above. An implementation MUST be able to take a strict interpretation on valid MSN windows and buffers being pre-supplied by the ULP. An implementation also MUST be able to layer cleanly over the LLP, where the DDP layer does not even receive the data until the LLP has decided that the segment/chunk is ackable. -- This is even a *reasonable* implementation over SCTP or when a general purpose TCP optimizing capability is in place. Therefore, a ULP MUST be defined such that each end can know the maximum number of untagged messages that could be in-flight. If this is not the case, then the Upper Layer Protocol requires more adaptation to use RDMAP/DDP. Any ULP that implicitly relied upon transport layer buffers to act as a flow control will require some adjustment -- but it is required. We cannot accept as a design goal an optimization for ULP behavior that would *break* a compliant implementation. Layered implemetnations are legal. Refusing to provide system buffering is legal. Therefore all ULPs MUST be defined to comply with these requirements -- which means they MUST include a method by which the Data Sink knows the maximum number of valid untagged messages that could be in flight. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 16 06:03:35 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA23772 for ; Wed, 16 Jul 2003 06:03:35 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cj7u-00070H-2R for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 06:03:11 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6GA3AA4026917 for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 06:03:10 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cj7t-000704-VJ for rddp-web-archive@optimus.ietf.org; Wed, 16 Jul 2003 06:03:09 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA23766 for ; Wed, 16 Jul 2003 06:03:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cj7q-0005P9-00 for rddp-web-archive@ietf.org; Wed, 16 Jul 2003 06:03:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19cj7k-0005P6-00 for rddp-web-archive@ietf.org; Wed, 16 Jul 2003 06:03:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cj7l-0006za-6n; Wed, 16 Jul 2003 06:03:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cj7b-0006zG-V9 for rddp@optimus.ietf.org; Wed, 16 Jul 2003 06:02:51 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA23747 for ; Wed, 16 Jul 2003 06:02:46 -0400 (EDT) From: pat_thaler@agilent.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cj7Y-0005P2-00 for rddp@ietf.org; Wed, 16 Jul 2003 06:02:48 -0400 Received: from msgbas1x.cos.agilent.com ([192.25.240.36]) by ietf-mx with esmtp (Exim 4.12) id 19cj7N-0005Oz-00 for rddp@ietf.org; Wed, 16 Jul 2003 06:02:37 -0400 Received: from relcos1.cos.agilent.com (relcos1.cos.agilent.com [130.29.152.239]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id 3968B15D91; Wed, 16 Jul 2003 04:02:31 -0600 (MDT) Received: from wcosbh22.cos.agilent.com (wcosbh22.cos.agilent.com [130.29.152.178]) by relcos1.cos.agilent.com (Postfix) with ESMTP id 6B9A4575; Wed, 16 Jul 2003 04:02:33 -0600 (MDT) content-class: urn:content-classes:message Subject: RE: [rddp] Some Minimal end-to-end Buffer Rules Required Date: Wed, 16 Jul 2003 04:02:30 -0600 Message-ID: <1BEBA5E8600DD4119A50009027AF54A0132E0438@axcs04.cos.agilent.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Thread-Topic: [rddp] Some Minimal end-to-end Buffer Rules Required X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 Thread-Index: AcNKJzrVCRYpnInnSL2AN6Tr54rnGQBV5ZKA To: , Cc: , Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable Caitlin, Your suggestion presumes that buffers are returned as they are emptied = by a process that knows which connection they were used by. In most = cases and especially in cases where one is designing for high = utilizations (and if one wasn't doing that one wouldn't need to use = DDP), that isn't likely to be the implementation. It would be common for = there to be some bulk process that gets freed buffers and restores them = to the shared queue or pool in bunches. It doesn't know or care which = connection the buffers were freed by. Also, this process is talking to = the shared queue manager part of the DDP implementation. It isn't = talking to the individual connections. A post buffer and grant credit operation would have to operate on both = the shared queue manager context and on the connection context which is = neither necessary or desireable. With bulk returns of buffers, the ULP's = buffer recycler would have to maintain a list of credits for connections = and give that list with the buffers to the DDP implementation which = would have to update multiple connection contexts.=20 It should be up to the application using the shared buffer queue to = manage the buffer pool for over usage. If higher independence between = connections is desired, then the application can use dedicated receiver = queues. Shared buffer queues are only one of several resources shared = between connections. I don't see any justification for your statement about virtually all = ULPs having limits on the number of untagged messages. In any case, if = the ULP does have such a limit, it can certainly see when its peer is = violating the limit and can disconnect. It doesn't need DDP to do that. Pat -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Monday, July 14, 2003 9:45 AM To: pat_thaler@agilent.com Cc: steph@cs.uchicago.edu; rddp@ietf.org Subject: RE: [rddp] Some Minimal end-to-end Buffer Rules Required On Monday, July 14, 2003, pat_thaler@agilent.com wrote: >This proposed rule below is not practical as the DDP implementation has = >no information on how many untagged messages the ULP has authorized. To = >add that information would require constant communication between the=20 >ULP and the DDP implementation to relay information about the credits=20 >that had been passed to the peer. Such communication would be a=20 >performance problem on the backplane. The ULP and not DDP should be=20 >responsible for detecting that the peer has overrun its credits.=20 > There are no more interactions than are required when fully provisioning the buffers. There is an inherent need to a) provide buffers and b) notify the ULP of completed reads. An example local interface would have the following capabilities: -- post buffer and credit to Endpoint Number of allowed untagged messages for Endpoint increments. Buffer is placed in receive queue for Endpoint (normally) or in an associated Shared Buffer Pool / Shared Receive Queue=20 (if the Endpoint is so configured). =20 -- grant credits to Endpoint without actually providing any buffers -- increasing the amount of underprovisioning. =20 -- post buffer without credit to Endpoint To decrease the number of buffers. =20 Only the first method/verb would be used during the processing of a typical transaction. A typical session would post some buffers, and if underprovisioning is to be used, some additional credits. As each completino is processed, a replacement buffer/credit is provided. Now the ULP is free to bundle these steps. If it wants to process three transactions and then post a single replacement with three buffers and credits. A more promising optimization might be to allow the replacement buffer and credit to be posted in the same operation as the reply message. I am not proposing a mandated procedure on *how* the ULP meters receives. Merely that it remain part of the loop. But given that the credit can easily be piggy-backed on re-enabling the buffer and/or sending the reply, I do not see how this can be a performance issue. =20 >What is necessary to protect other connections in a shared buffer pool=20 >situation is to deal with the problem of a peer that has tied up=20 >excessive numbers of partially filled buffers. The ULP can only see=20 >completions, so an attack is possible by sending segments for many=20 >messages without completing any message. For instance, in the verbs=20 >document, we put an alarm (with threshold settable by the ULP) for=20 >excessive incomplete buffers in a receive queue. An Alarm that requires a ULP response is inherently suspect as a defense againt a Denial of Service attack. A static limit also does nothing to prevent an over-eager peer that always puts the next request on the wire too early. A static limit is fixed at N, it does not realize that it should be N-m because there are already m completed messages being processed. Virtually every ULP protocol seen to data limits the number of untagged messages that have not been responded to. Expanding the range of Valid MSN by posting a nrw receive operation (whether to the specific EP or to its pool) accomplishes this because it is done *after* the ULP hass finished processing a message. This sets the valid MSN window to the correct value, and can do so as part of inherent per-transaction interactions. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 16 09:56:35 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA00410 for ; Wed, 16 Jul 2003 09:56:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cmlN-0004KM-36 for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 09:56:09 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6GDu9TD016630 for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 09:56:09 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cmlN-0004K9-05 for rddp-web-archive@optimus.ietf.org; Wed, 16 Jul 2003 09:56:09 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA00364 for ; Wed, 16 Jul 2003 09:56:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cmlK-0007Kg-00 for rddp-web-archive@ietf.org; Wed, 16 Jul 2003 09:56:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19cmlF-0007Kb-00 for rddp-web-archive@ietf.org; Wed, 16 Jul 2003 09:56:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cmlG-0004Gx-K9; Wed, 16 Jul 2003 09:56:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cmkh-0004Fs-AT for rddp@optimus.ietf.org; Wed, 16 Jul 2003 09:55:27 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA00351 for ; Wed, 16 Jul 2003 09:55:22 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cmkf-0007K0-00 for rddp@ietf.org; Wed, 16 Jul 2003 09:55:25 -0400 Received: from e33.co.us.ibm.com ([32.97.110.131]) by ietf-mx with esmtp (Exim 4.12) id 19cmkK-0007J6-00 for rddp@ietf.org; Wed, 16 Jul 2003 09:55:04 -0400 Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e33.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h6GDr6j3229680 for ; Wed, 16 Jul 2003 09:53:06 -0400 Received: from d03nm115.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h6GDr4dG181250 for ; Wed, 16 Jul 2003 07:53:05 -0600 To: rddp@ietf.org MIME-Version: 1.0 Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 From: John Hufferd Message-ID: Date: Wed, 16 Jul 2003 06:52:58 -0700 X-MIMETrack: Serialize by Router on D03NM115/03/M/IBM(Release 6.0.1 [IBM]|June 10, 2003) at 07/16/2003 07:53:04, Serialize complete at 07/16/2003 07:53:04 Content-Type: multipart/alternative; boundary="=_alternative 004C269088256D65_=" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multipart message in MIME format. --=_alternative 004C269088256D65_= Content-Type: text/plain; charset="us-ascii" Again, I think that Mallikarjun stated things correctly. There are important applications which can only correctly predict the number of buffers needed in non error/special/emergency conditions. When error/special/emergency conditions occur, the number and urgency of the handling is problematical. So rather than going back through all the arguments yet again, please accept that these are real and important applications that want to exploit RDMA. Both the shared buffer and non shared buffer case can use some sort of graceful handling technique. When we talk through this set of issues, in the past, with several different RNIC vendors, the various RNIC vendors had their own specific solutions (each one different from the others). The common aspect was that their solutions did not reflect through to the ULP and were good enough to permit system glitches, or special dispatching of higher level tasks etc. to occur, yet finally permitting the ULP to get around to posting the needed buffers. Now there is another point that I have been having a problem with in the various reflector messages, and that is: I do not think it is useful to call the types of message that can use special handling, to be unauthorized messages. They are not. They are only unpredictable. If the ULP does not, in due time, post the approprate buffers, the connection will of course be brought down. That should not be an issue here. I also do not think it is approprate to compare the advertisement of buffers (STags) for use with RDMA Reads and Writes with the authorization of untagged messages. the RDMA Reads/Writes are generally used for sending agreed upon data buffers, however, RDMA Send (untagged) messages is often used for control, and special handling messages. And it is the handling of unexpected, but permitted "special handling" messages which we have been attempting to address. So the only thing the system vendors are asking for is NOT to include language in the Draft that would prevent them from being able to select a RNIC vendor which can meet their systems' needs. . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/System Group, San Jose CA Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Alt Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com --=_alternative 004C269088256D65_= Content-Type: text/html; charset="us-ascii"
Again, I think that Mallikarjun stated things correctly.  There are important applications which can only correctly predict the number of buffers needed in non error/special/emergency conditions.  When error/special/emergency conditions occur, the number and urgency of the handling is problematical.

So rather than going back through all the arguments yet again, please accept that these are real and important applications that want to exploit RDMA.

Both the shared buffer and non shared buffer case can use some sort of graceful handling technique.  When we talk through this set of issues, in the past, with several different RNIC vendors, the various RNIC vendors had their own specific solutions (each one different from the others).  The common aspect was that their solutions did not reflect through to the ULP and were good enough to permit system glitches, or special dispatching of higher level tasks etc. to occur, yet finally permitting the ULP to get around to posting the needed buffers.

Now there is another point that I have been having a problem with in the various reflector messages, and that is: I do not think it is useful to call the types of message that can use special handling, to be unauthorized messages.  They are not.  They are only unpredictable.  If the ULP does not, in due time, post the approprate buffers, the connection will of course be brought down.  That should not be an issue here.  

I also do not think it is approprate to compare the advertisement of buffers (STags) for use with RDMA Reads and Writes with the authorization of untagged messages.  the RDMA Reads/Writes are generally used for sending agreed upon data buffers, however, RDMA Send (untagged) messages is often used for control, and special handling messages.  And it is the handling of unexpected, but permitted "special handling" messages which we have been attempting to  address.

So the only thing the system vendors are asking for is NOT to include language in the Draft that would prevent them from being able to select a RNIC vendor which can meet their systems' needs.


.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: hufferd@us.ibm.com
--=_alternative 004C269088256D65_=-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 16 10:11:58 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA02102 for ; Wed, 16 Jul 2003 10:11:58 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cn0G-0005Jz-PZ for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 10:11:33 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6GEBWfE020451 for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 10:11:32 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cn0G-0005Jm-Ml for rddp-web-archive@optimus.ietf.org; Wed, 16 Jul 2003 10:11:32 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA02035 for ; Wed, 16 Jul 2003 10:11:27 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cmzl-0005HN-HP; Wed, 16 Jul 2003 10:11:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cmzW-0005Gt-LG for rddp@optimus.ietf.org; Wed, 16 Jul 2003 10:10:46 -0400 Received: from thebe.your-site.com (thebe.your-site.com [140.186.45.26]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA01907 for ; Wed, 16 Jul 2003 10:10:41 -0400 (EDT) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 4CE392452D8; Wed, 16 Jul 2003 10:10:19 -0400 (EDT) Date: Wed, 16 Jul 2003 09:10:10 -0500 From: Caitlin Bestler Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required To: John Hufferd Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Wednesday, July 16, 2003, hufferd@us.ibm.com wrote: >Again, I think that Mallikarjun stated things correctly. There are >important applications which can only correctly predict the number of >buffers needed in non error/special/emergency conditions. When >error/special/emergency conditions occur, the number and urgency of the >handling is problematical. > >So rather than going back through all the arguments yet again, >please accept that these are real and important applications >that want to exploit RDMA. Then these applications do not work over InfiniBand? You have a credit limit over IB RC, and it is even less flexible than what you can do with iWARP. If these applications MUST send an unregulated number of messages on demand then they should be using an unreliable transport. More likely, the applications as they currently exist are used to being throttled transparently by LLP flow control. (or built into the RDMA layer itself, as with InfiniBand). That does not work for iWARP. This was one of the early design decisions. Because middleboxes can generate LLP Acks that do not actually imply DDP delivery the only way to have InfiniBand style credits would be to generate *additional* network traffic at the RDMAP or DDP layer. This was properly and resoundingly rejected. But there was never a concept of *eliminating* credits, merely making their enforcement local. Rather than receiving credits from your peer, you receive credits when the local ULP posts a receive operation (which also acts as a throttle under IB). Bottom line: there are valid implementations of RDMA which *cannot* forgive invalid MSNs. The ULPs MUST be designed to work under these circumtances. If that condition is met, then individual implemenations may choose to be more fault tolerant. But it is unacceptable to consider as a factor ULP behavior that will break a *compliant* RNIC. Which means that invalid MSNs MUST be rejected. How to handle valid MSNs without a buffer is a local issue. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 16 10:20:33 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA02870 for ; Wed, 16 Jul 2003 10:20:33 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cn8Z-0005oU-Jv for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 10:20:08 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6GEK7Vv022346 for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 10:20:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cn8Z-0005oL-FE for rddp-web-archive@optimus.ietf.org; Wed, 16 Jul 2003 10:20:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA02838 for ; Wed, 16 Jul 2003 10:20:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cn8X-0007bb-00 for rddp-web-archive@ietf.org; Wed, 16 Jul 2003 10:20:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19cn8R-0007bY-00 for rddp-web-archive@ietf.org; Wed, 16 Jul 2003 10:19:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cn8T-0005mc-3Q; Wed, 16 Jul 2003 10:20:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cn7w-0005le-RK for rddp@optimus.ietf.org; Wed, 16 Jul 2003 10:19:28 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA02788 for ; Wed, 16 Jul 2003 10:19:23 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cn7r-0007b6-00 for rddp@ietf.org; Wed, 16 Jul 2003 10:19:23 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19cn7g-0007aY-00 for rddp@ietf.org; Wed, 16 Jul 2003 10:19:12 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 40BDB24512A; Wed, 16 Jul 2003 10:18:18 -0400 (EDT) Date: Wed, 16 Jul 2003 09:18:09 -0500 From: Caitlin Bestler Subject: RE: [rddp] Some Minimal end-to-end Buffer Rules Required To: pat_thaler@agilent.com Cc: steph@cs.uchicago.edu, rddp@ietf.org X-Priority: 3 In-Reply-To: <1BEBA5E8600DD4119A50009027AF54A0132E0438@axcs04.cos.agilent.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Answering different portions in an arbitrary order... On Wednesday, July 16, 2003, pat_thaler@agilent.com wrote: >I don't see any justification for your statement about virtually all >ULPs having limits on the number of untagged messages. Can you cite a published protocol that works over RDMA that does *not* have a concept of message credits? Every protocol over IB has credits. DAFS has credits. NFS over RDMA has credits. If the ULP does not have limits on the number of untagged messages, how is it doing flow control? Keep in mind that the ULP does not see tagged messages, and has no way of knowing how much tagged traffic was actually done associated with any untagged message (other than the presumption being that it was what was allowed, but technically the peer is allowed to send redundant tagged writes as long as it doesn't expect them to be processed in a specific order). The implications of RDMA on application flow control are covered in the applicability draft. If you have questions, why not review that? My position remains that you cannot correctly specify an Upper Layer Protocol to run over iWARP without explicit control over the number of in-flight untagged messages. Therefore the local consumer *knows* how many untagged messages to allow. It is only a matter of how to share that data. > In any case, if the ULP does have such a limit, it can > certainly see when its peer is violating the limit and can > disconnect. It doesn't need DDP to do that. That was already covered in prior postings on this topic, to which nobody has offered a counter-argument. To recap, the local ULP is not in a position to deal with either of the extremes: -- A typical design will place completions on an asynchronous and opaque queue, which will frequently be shared across multiple streams. The only way for the application to know that there are N messages from stream X is to completely drain the queue. That takes time. The probability that a peer could consistently send its Nth reply *before* being authorized to do so is very high. "Off by one" errors are very common. "Off by one" errors that *benefit* the client are not likely to be caught. Once this edge is discovered, all clients will feel competitive pressure to emultate this "error", some of those emulations will have "off by one" errors and actually be *two* over the limit... -- A Denial of Service attack succeeds if it requires the ULP to take action. But these both miss the critical point. If the MSN is invalid it should be rejected as early as possible. The DDP layer rejects invalid STags and invalid TOs. It does not wait for the STag to become valid again (even though it typically will). Why should it wait for the MSN to become valid and then deliver it? As I pointed out, before Shared Receive Queues were proposed, there was no doubt in any discusions or any documents that the MSN would be validated by the Endpoint/QP on arriving messages. >Your suggestion presumes that buffers are returned as they are emptied >by a process that knows which connection they were used by. In most >cases and especially in cases where one is designing for high >utilizations (and if one wasn't doing that one wouldn't need to use >DDP), that isn't likely to be the implementation. The software component that processes the contents of a buffer is not aware of which connection the data was delivered on? What does it think the data is? Who will it tell that the data was good / bad? Is it accepting an indefinite/infinite supply of such messages from anybody? That sounds like an unreliable protocol. DDP messages are delivered over a reliable transport, and rely upon ULP flow control. How can the ULP do flow control if it has forgotten who it is talking with? All RDMA-specific APIs that I am aware of preserve this information. Even when using a central completion queue, each delivered untagged message will note the Endpoint/QP that it came from. >It would be common for there to be some bulk process that gets >freed buffers and restores them to the shared queue or pool in >bunches. It doesn't know or care which connection the buffers >were freed by. Also, this process is talking to the shared >queue manager part of the DDP implementation. It isn't talking >to the individual connections. > If you are running a shared pool the buffers will be a uniform size. Otherwise you are running a true heap, something that I have trouble imagining being stable under a continuous multi-GByte allocation demand. Even simple real-time system avoid dynamic variable allocation, I would assume that virtually all RNIC designs will be fetching a buffer of a fixed size (whether from a pool or an ordered receive queue) and then expecting the message to match. The fanciest I can imagine is that there might be multiple size dependent pools. In any event, the most common technique will be to pre-stock the pool and then enter a loop to: - receive a request as a completion of an untagged receive. - process the request, sending any tagged messages. - post the replacement receive buffer (i.e. the same one) - post the untagged message that is the reply. There is no real benefit to coalescing them before placing them in the common pool. And the natural time to do so would be when completing the handling of each untagged receive completion. In any event, I did not propose that the receive credit had to be supplied with a buffer. Those operations can be seperated. It could just as easily be attached to sending the reply. >A post buffer and grant credit operation would have to operate on both >the shared queue manager context and on the connection context which is >neither necessary or desireable. A Work Request is posted to the Endpoint that optionally supplies a buffer. The Endpoint increments its credits, and then posts the buffer either to its own receive queue or to a shared buffer pool (according to the Endpoint's configuration). The link between the endpoint and the shared buffer pool already exists. It is used to allow the Endpoint to *allocate* from the pool when an untagged message is detected. The allocation must be performed at peak rates to match the link on demand, the restore operation can be done in slack time at a sustained rate. It is *easier* than allocating on demand. The mechanisms are obvously in place to deal with serializing allocation requests? Why does this become undesirable when restocking the buffer pool? Keep in mind that the shared buffer pool algorithm I have described allows the ULP to be designed without knowing whether buffer-pooling is in use or not, and keeps the per-transactin processing loop virtually identical whether using a buffer-pool or a receive queue. >With bulk returns of buffers, the ULP's >buffer recycler would have to maintain a list of credits for connections >and give that list with the buffers to the DDP implementation which >would have to update multiple connection contexts. > No, it just has to remember which Endpoint/QP is having the credit replenished. The Endpoint/QP already knows what pool it belongs to. It is never more than a single Endpoint/QP that is being credited, and never more than a single buffer pool or receive queue that is being restocked. >It should be up to the application using the shared buffer queue to >manage the buffer pool for over usage. If higher independence between >connections is desired, then the application can use dedicated receiver >queues. Shared buffer queues are only one of several resources shared >between connections. > Which is why accepting invalid MSNs is particularly dangerous. If the completion queue is defined more narrowly than the buffer pool, the excess completions from invalid MSNs could cause a CQ overflow. I have not seen a local interface that can gracefully recover from a CQ overflow. One or more streams will have to be aborted. It is factors like that which triggered my concerns. Accepting invalid MSNs opens multiple vunerabilities to DoS attacks, errors and client abuse of server resources. Buffer pooling is desirable, but can be achieved *without* changing the end-to-end credit semantics or creating any additional security vunerabilities. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 16 10:35:58 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA03915 for ; Wed, 16 Jul 2003 10:35:58 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cnNV-0006lj-BH for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 10:35:33 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6GEZXub026014 for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 10:35:33 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cnNV-0006lV-6n for rddp-web-archive@optimus.ietf.org; Wed, 16 Jul 2003 10:35:33 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA03894 for ; Wed, 16 Jul 2003 10:35:27 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cnMz-0006hk-2E; Wed, 16 Jul 2003 10:35:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cnME-0006h2-FS for rddp@optimus.ietf.org; Wed, 16 Jul 2003 10:34:14 -0400 Received: from thebe.your-site.com (thebe.your-site.com [140.186.45.26]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA03728 for ; Wed, 16 Jul 2003 10:34:08 -0400 (EDT) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 594C0245135; Wed, 16 Jul 2003 10:33:44 -0400 (EDT) Date: Wed, 16 Jul 2003 09:33:35 -0500 From: Caitlin Bestler Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required To: John Hufferd Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Wednesday, July 16, 2003, hufferd@us.ibm.com wrote: >So the only thing the system vendors are asking for is NOT to >include language in the Draft that would prevent them from >being able to select a RNIC vendor which can meet their >systems' needs. But the draft MUST include language that allows the design of upper layer protocols in an implementation independent fashion. I do not see how that is possible if one peer is allowed to send an "unpredictable" number of untagged messages. The receiving side MUST be able to bound this problem. The availability of buffers is not the critical issue, but rather sizing of completion queues and the resources required to track in-process untagged messages. You cannot have ULP flow control if one of the sides is ignorant on how many untagged messages can be in-flight. LLP flow control does not work. It would be seriously incorrect to use LLP buffer flow control to hold off *all* DDP Segments because there is no buffer space for an *untagged* message. That gets even worse under SCTP where you would have to hold off *all* messages on *all* streams because *one* stream did not not have an available untagged buffer. It's even more complex: you could accept *some* untagged segments because they are completing in-process messages, but not others. For Endpoints with a simple receive queue, buffers are posted for specific MSNs. Clearly LLP flow control cannot allow only those packets with Tagged segments or Untagged segments for one of the following MSNs. Can you propose what it is that you expect of an RNIC that would enable a ULP to send an unpredictable number of untagged messages in a reliable, implementation independent, fashion? If there is a draft of a ULP that makes such assumptions, it needs to be corrected. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 16 10:40:57 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA04276 for ; Wed, 16 Jul 2003 10:40:57 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cnSK-0007NC-Ct for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 10:40:32 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6GEeW0L028341 for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 10:40:32 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cnSK-0007N2-2X for rddp-web-archive@optimus.ietf.org; Wed, 16 Jul 2003 10:40:32 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA04254 for ; Wed, 16 Jul 2003 10:40:26 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cnRp-0007Ko-AV; Wed, 16 Jul 2003 10:40:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cnRa-0007K6-St for rddp@optimus.ietf.org; Wed, 16 Jul 2003 10:39:46 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA04204 for ; Wed, 16 Jul 2003 10:39:41 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cnRY-00001x-00 for rddp@ietf.org; Wed, 16 Jul 2003 10:39:44 -0400 Received: from e35.co.us.ibm.com ([32.97.110.133]) by ietf-mx with esmtp (Exim 4.12) id 19cnRD-000005-00 for rddp@ietf.org; Wed, 16 Jul 2003 10:39:23 -0400 Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e35.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h6GEaIc8194126; Wed, 16 Jul 2003 10:36:18 -0400 Received: from d03nm115.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h6GEaCdG184152; Wed, 16 Jul 2003 08:36:12 -0600 To: Caitlin Bestler Cc: rddp@ietf.org MIME-Version: 1.0 Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 From: John Hufferd Message-ID: Date: Wed, 16 Jul 2003 07:35:55 -0700 X-MIMETrack: Serialize by Router on D03NM115/03/M/IBM(Release 6.0.1 [IBM]|June 10, 2003) at 07/16/2003 08:36:12, Serialize complete at 07/16/2003 08:36:12 Content-Type: multipart/alternative; boundary="=_alternative 00502A4688256D65_=" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multipart message in MIME format. --=_alternative 00502A4688256D65_= Content-Type: text/plain; charset="us-ascii" Responses in line [Huff/] ... [/Huff] . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/System Group, San Jose CA Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Alt Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com Caitlin Bestler 07/16/2003 07:10 AM To: John Hufferd/San Jose/IBM@IBMUS cc: rddp@ietf.org Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required On Wednesday, July 16, 2003, hufferd@us.ibm.com wrote: >Again, I think that Mallikarjun stated things correctly. There are >important applications which can only correctly predict the number of >buffers needed in non error/special/emergency conditions. When >error/special/emergency conditions occur, the number and urgency of the >handling is problematical. > >So rather than going back through all the arguments yet again, >please accept that these are real and important applications >that want to exploit RDMA. Then these applications do not work over InfiniBand? [Huff/] These application have not yet been shown to operate on InfiniBand, for a number of reasons that are independent of what we have been talking about here. On the other hand, there are currently extensions for InfiniBand being proposed which may remove these other restrictions. So we will have to wait and see about that. [\Huff] You have a credit limit over IB RC, and it is even less flexible than what you can do with iWARP. If these applications MUST send an unregulated number of messages on demand then they should be using an unreliable transport. [Huff/] Now that is not a helpful response. [\Huff] More likely, the applications as they currently exist are used to being throttled transparently by LLP flow control. (or built into the RDMA layer itself, as with InfiniBand). That does not work for iWARP. This was one of the early design decisions. Because middleboxes can generate LLP Acks that do not actually imply DDP delivery the only way to have InfiniBand style credits would be to generate *additional* network traffic at the RDMAP or DDP layer. This was properly and resoundingly rejected. [Huff/] I think we are drifting off topic here. [\Huff] But there was never a concept of *eliminating* credits, merely making their enforcement local. Rather than receiving credits from your peer, you receive credits when the local ULP posts a receive operation (which also acts as a throttle under IB). [Huff/] Again off topic [\Huff] Bottom line: there are valid implementations of RDMA which *cannot* forgive invalid MSNs. The ULPs MUST be designed to work under these circumtances. If that condition is met, then individual implemenations may choose to be more fault tolerant. But it is unacceptable to consider as a factor ULP behavior that will break a *compliant* RNIC. [Huff/] I have not, in any of my messages suggested that we need to have a solution that breaks compliant RNICs. The point was always to define what "compliant" meant. You have been building up a strawman and knocking it down. We should get back to the original points detailed by Mallikarjun about ensuring that the language in the specification is such that the requirements of the various System Vendors can be met. [\Huff] Which means that invalid MSNs MUST be rejected. How to handle valid MSNs without a buffer is a local issue. --=_alternative 00502A4688256D65_= Content-Type: text/html; charset="us-ascii"
Responses in line [Huff/] ... [/Huff]
.
.
John L. Hufferd
Senior Technical Staff Member (STSM)
IBM/System Group, San Jose CA
Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688
Alt Office: (408) 997-6136, Cell: (408) 499-9702
Internet Address: hufferd@us.ibm.com


Caitlin Bestler <cait@asomi.com>

07/16/2003 07:10 AM

       
        To:        John Hufferd/San Jose/IBM@IBMUS
        cc:        rddp@ietf.org
        Subject:        Re: [rddp] Some Minimal end-to-end Buffer Rules Required



On Wednesday, July 16, 2003, hufferd@us.ibm.com wrote:

>Again, I think that Mallikarjun stated things correctly.  There are
>important applications which can only correctly predict the number of
>buffers needed in non error/special/emergency conditions.  When
>error/special/emergency conditions occur, the number and urgency of the
>handling is problematical.
>
>So rather than going back through all the arguments yet again,
>please accept that these are real and important applications
>that want to exploit RDMA.

Then these applications do not work over InfiniBand?

[Huff/]
These application have not yet been shown to operate on InfiniBand, for a number of reasons that are independent of what we have been talking about here.  On the other hand, there are currently extensions for InfiniBand being proposed which may remove these other restrictions.  So we will have to wait and see about that.
[\Huff]

You have
a credit limit over IB RC, and it is even less flexible than
what you can do with iWARP.

If these applications MUST send an unregulated number of messages
on demand then they should be using an unreliable transport.

[Huff/]
Now that is not a helpful response.
[\Huff]

More likely, the applications as they currently exist are used to
being throttled transparently by LLP flow control. (or built into
the RDMA layer itself, as with InfiniBand).

That does not work for iWARP. This was one of the early design
decisions. Because middleboxes can generate LLP Acks that do not
actually imply DDP delivery the only way to have InfiniBand style
credits would be to generate *additional* network traffic at
the RDMAP or DDP layer. This was properly and resoundingly
rejected.

[Huff/]
I think we are drifting off topic here.
[\Huff]

But there was never a concept of *eliminating* credits, merely
making their enforcement local. Rather than receiving credits
from your peer, you receive credits when the local ULP posts
a receive operation (which also acts as a throttle under IB).
[Huff/] Again off topic [\Huff]

Bottom line: there are valid implementations of RDMA which
*cannot* forgive invalid MSNs. The ULPs MUST be designed to
work under these circumtances. If that condition is met, then
individual implemenations may choose to be more fault tolerant.
But it is unacceptable to consider as a factor ULP behavior
that will break a *compliant* RNIC.

[Huff/]
I have not, in any of my messages suggested that we need to have a solution that breaks compliant RNICs.  The point was always to define what "compliant" meant.  You have been building up a strawman and knocking it down.  We should get back to the original points detailed by Mallikarjun about ensuring that the language in the specification is such that the requirements of the various System Vendors can be met.
[\Huff]

Which means that invalid MSNs MUST be rejected. How to handle
valid MSNs without a buffer is a local issue.

--=_alternative 00502A4688256D65_=-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 16 12:10:34 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA07106 for ; Wed, 16 Jul 2003 12:10:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cor2-0004iS-Eu for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 12:10:08 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6GGA8GS018123 for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 12:10:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19cor2-0004iE-Bu for rddp-web-archive@optimus.ietf.org; Wed, 16 Jul 2003 12:10:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA07098 for ; Wed, 16 Jul 2003 12:10:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19cor1-0000p2-00 for rddp-web-archive@ietf.org; Wed, 16 Jul 2003 12:10:07 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19coqv-0000oz-00 for rddp-web-archive@ietf.org; Wed, 16 Jul 2003 12:10:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19coqv-0004hf-6f; Wed, 16 Jul 2003 12:10:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19coqP-0004hG-M2 for rddp@optimus.ietf.org; Wed, 16 Jul 2003 12:09:35 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA07079 for ; Wed, 16 Jul 2003 12:09:23 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19coqM-0000ol-00 for rddp@ietf.org; Wed, 16 Jul 2003 12:09:26 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19coqC-0000oh-00 for rddp@ietf.org; Wed, 16 Jul 2003 12:09:16 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id DEB51244E21; Wed, 16 Jul 2003 12:08:30 -0400 (EDT) Date: Wed, 16 Jul 2003 11:08:22 -0500 From: Caitlin Bestler To: rddp@ietf.org Cc: John Hufferd , pat_thaler@agilent.com, "Mallikarjun C." X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] Untagged Message Validation Recap Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Initial drafts required that the local ULP pre-post a buffer on a stream specific basis for each authorized untagged message. This is an unreasonable requirement that can have a serious impact on servers that support a large number of clients concurrently. There is full agreement on this point. I'll also point out for the record that I raised the first objection on this point when David issued a last call on the architecture document. The current drafts still imply that the abence of a buffer requires that the stream be terminated. This is excessive. There is full agreement on this point. At the minimum it must be clear that receipt of a valid MSN for which there is no buffer is a local fault. Not only is the local implementatin *allowed* to recover from such a fault, it is in fact commendable to do so. That leaves the question of what to do with Invalid MSNs. What I propose here are two rules: 1) ULP design MUTS NOT rely upon delivery of invalid MSNs. Through *some* mechanism, the Data Sink ULP must be able to predict the maximum number of untagged messages that may be in flight at any given instant. Note that this is true even if the DDP layer is ignorant of this limitation. The ULP is responsible for properly sizing the Completion Queue. Untagged messages generate completions. You cannot reliably size the Completion Queue if you do not know how many messages may be in flight. In terms of the verbs draft - I am stating that it is incorrect and unacceptable for the Upper Layer Protocol to assume that a Shared Receive Queue, or shared Buffers are in use. The Upper Layer Protocol MUST be designed so that a compliant implementation is possible with simple Receive Queues. That requires a shared understanding between the ULP peers as to the maximum number of in-flight untagged messages. There is no escaping that. How can the ULP be responsible for flow control if it isn't paying any attention? 2) Accepting an untagged message with an invalid MSN MUST NOT place at risk the reception of any untagged message with a valid MSN. Allowing Client X's violation of the ULP to disrupt Client Y is clearly a major security vunerability. The verbs draft makes this a serious vunerability by actually requiring the DDP layer to continue processing MSNs until the ULP pulls the plug. A properly designed attack could generate a burst of untagged message completions which would overwhelm the Completion Queue and drain the buffer pool. Is it reasonable for the DDP layer to enforce validity of MSNs? I believe that it is easy and natural for the local ULP to convey the current range of valid MSNs to the Endpoint. In fact in the initial drafts this was the assumption. I find claims that this is now somehow overly burdensome to be unconvincing. But specifically: The same "post receive operation" used with simple Receive Queues could be used to grant the credit and then populate a shared Buffer Pool. The link between Endpoint and Buffer Pool must exist upon message arrival, so it is there when Work Requests are processed as well. Typically credits are communiciated to the peer via an untagged message acting as a reply (imlicitly, as with DAFS, or explicitly as with NFS over RDMA). An option to increment the untagged receive credits attached to an untagged send would allow per Endpoint/QP credit tracking wihtout tying it to buffer replenishment or requiring any additional crossing of the system bus. There are reasons why the ULP is in a very poor position to enforce MSN validation: Messages with invalid MSNs must be accepted provisionally. Provisionally they consume resources, both buffers and CQEs. This may result in CQ overflow and/or buffer exhaustion *before* the ULP has a chance to respond. ULP diagnosis of MSN validity is asynchronous. The ULP cannot tell if an MSN that is valid by the time the ULP examines it had in fact "jumped the gun" and was sent before it had been authorized. If the ULP has decided to limit a client to 4 in-flight messages, it is important that they not be able to send 5. But it is next to impossible for the ULP to determine whether then nth request was sent before or after the n-5th reply had been sent. Having every client round up the number of in-flight messages allowed by one would have a serious impact on a multi-client server. In summary, underprovisioning is fine. It is even a good idea when done in a way that buffers shortfalls occur very rarely. ULPs that want to abdicate their responsibility to implement flow control are not fine. They cannot rely upon the local implementations to sweep such an abdication under the rug. Many non-RDMA ULPs rely upon the LLP to provide flow control for them. It is a beneficial side-effect of the undersirable bottleneck of traditional LLPs. RDMA eliminates that bottleneck. That means that ULPs will have to actually think about how they are flow controlled. Review the applicability draft and raise questions about the analysis there if you disagree. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 16 16:52:41 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA16970 for ; Wed, 16 Jul 2003 16:52:41 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ctG3-0004kz-1u for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 16:52:15 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6GKqFU7018280 for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 16:52:15 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ctG1-0004kl-I5 for rddp-web-archive@optimus.ietf.org; Wed, 16 Jul 2003 16:52:13 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA16967 for ; Wed, 16 Jul 2003 16:52:08 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ctFz-00039f-00 for rddp-web-archive@ietf.org; Wed, 16 Jul 2003 16:52:11 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ctFt-00039a-00 for rddp-web-archive@ietf.org; Wed, 16 Jul 2003 16:52:05 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ctFp-0004j4-7h; Wed, 16 Jul 2003 16:52:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ctFF-0004ir-IZ for rddp@optimus.ietf.org; Wed, 16 Jul 2003 16:51:27 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA16956 for ; Wed, 16 Jul 2003 16:51:20 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ctFD-00039A-00 for rddp@ietf.org; Wed, 16 Jul 2003 16:51:23 -0400 Received: from e33.co.us.ibm.com ([32.97.110.131]) by ietf-mx with esmtp (Exim 4.12) id 19ctEs-00038N-00 for rddp@ietf.org; Wed, 16 Jul 2003 16:51:02 -0400 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e33.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h6GKnsj3261916; Wed, 16 Jul 2003 16:49:54 -0400 Received: from d03nm115.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay02.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h6GKngMb157676; Wed, 16 Jul 2003 14:49:43 -0600 To: Caitlin Bestler Cc: "Mallikarjun C." , pat_thaler@agilent.com, rddp@ietf.org X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 Message-ID: From: John Hufferd Date: Wed, 16 Jul 2003 13:41:03 -0700 X-MIMETrack: Serialize by Router on D03NM115/03/M/IBM(Release 6.0.1 [IBM]|June 10, 2003) at 07/16/2003 14:49:54 MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Subject: [rddp] Re: Untagged Message Validation Recap Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Caitlin, I do not think your characterization of the proposal is correct or fair. No one has said what you are trying to read into our messages! The point is that the various protocols we are talking about, actually have what most folks think are an adequate number of pre-authorized buffers. But just as we talked about system impacts that were out of the control of the application when shared Buffers/RQs are used, this same situation occurs for Single RQs. That is, an application might have a contingency of say 4 extra buffers, which work in all the testing and a number of various systems. As long as the local system is acting "normal", then even if the remote system has a trauma, there will be enough buffers queued so that the ULP can receive the message and then queue the next request. But sometimes that will not always be the case. This is not due to any fault of the ULP. It is often caused by freakish events happening in the local system as the remote system is having a trauma which it dearly needs to report (hence why the ULP had sum extra buffers defined, just in case). Also sometimes, the thing that is causing the problem may be also happening when, or even causing a lot of out-of-order arrival of a number of these messages to the ULP. The out-of-order arrival causes many of the WQE buffers to be tied up but not completed. As pointed out by Mallikarjun and Pat, these corner cases exist, and through no fault of the ULP. System vendors would like to shield their customers from these types of disappointments. As I have stated before a number of different RNIC vendors are planning on doing a number of things which might give the system some latitude. However, that is all it is, some latitude. These various implementations still require the ULP to function as we have been stating but they can do so based on "ULP time" (as opposed to "System time"), with the RI and the RNIC making up the difference. There has been no statements made by myself, Mallikarjun, or Pat that said that they want the ULP not to have to post the approprate number of buffers and replenish them as approprate. We have just said that we need the words in the Draft to permit agreements between the System vendors and the RNIC vendors for what we call graceful handling of these Rare situations. We all understand that this flexibility in the system and RNIC, is in fact a very rare and seldom used capability, but as it turns out, the flexibility permits the system vendors to cover over, what in a narrow view may seem like, their various erratic behaviors, and permit the ULP to actually be more portable across different systems. In the various implementations that the vendors have proposed, all sooner or later ended up killing the QP, if the situation was not adequately handled by the ULP. And in no implementation that I saw, was any other QP in jeopardy. These are system issues, NOT ULP issues! And the system needs ways to handle it. Hence the request for loser language. . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/System Group, San Jose CA Main Office: (408) 256-0403, Tie: 276-0403, eFax: (408) 904-4688 Alt Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com Caitlin Bestler To: rddp@ietf.org cc: John Hufferd/San Jose/IBM@IBMUS, pat_thaler@agilent.com, "Mallikarjun C." 07/16/2003 09:08 AM Subject: Untagged Message Validation Recap Initial drafts required that the local ULP pre-post a buffer on a stream specific basis for each authorized untagged message. This is an unreasonable requirement that can have a serious impact on servers that support a large number of clients concurrently. There is full agreement on this point. I'll also point out for the record that I raised the first objection on this point when David issued a last call on the architecture document. The current drafts still imply that the abence of a buffer requires that the stream be terminated. This is excessive. There is full agreement on this point. At the minimum it must be clear that receipt of a valid MSN for which there is no buffer is a local fault. Not only is the local implementatin *allowed* to recover from such a fault, it is in fact commendable to do so. That leaves the question of what to do with Invalid MSNs. What I propose here are two rules: 1) ULP design MUTS NOT rely upon delivery of invalid MSNs. Through *some* mechanism, the Data Sink ULP must be able to predict the maximum number of untagged messages that may be in flight at any given instant. Note that this is true even if the DDP layer is ignorant of this limitation. The ULP is responsible for properly sizing the Completion Queue. Untagged messages generate completions. You cannot reliably size the Completion Queue if you do not know how many messages may be in flight. In terms of the verbs draft - I am stating that it is incorrect and unacceptable for the Upper Layer Protocol to assume that a Shared Receive Queue, or shared Buffers are in use. The Upper Layer Protocol MUST be designed so that a compliant implementation is possible with simple Receive Queues. That requires a shared understanding between the ULP peers as to the maximum number of in-flight untagged messages. There is no escaping that. How can the ULP be responsible for flow control if it isn't paying any attention? 2) Accepting an untagged message with an invalid MSN MUST NOT place at risk the reception of any untagged message with a valid MSN. Allowing Client X's violation of the ULP to disrupt Client Y is clearly a major security vunerability. The verbs draft makes this a serious vunerability by actually requiring the DDP layer to continue processing MSNs until the ULP pulls the plug. A properly designed attack could generate a burst of untagged message completions which would overwhelm the Completion Queue and drain the buffer pool. Is it reasonable for the DDP layer to enforce validity of MSNs? I believe that it is easy and natural for the local ULP to convey the current range of valid MSNs to the Endpoint. In fact in the initial drafts this was the assumption. I find claims that this is now somehow overly burdensome to be unconvincing. But specifically: The same "post receive operation" used with simple Receive Queues could be used to grant the credit and then populate a shared Buffer Pool. The link between Endpoint and Buffer Pool must exist upon message arrival, so it is there when Work Requests are processed as well. Typically credits are communiciated to the peer via an untagged message acting as a reply (imlicitly, as with DAFS, or explicitly as with NFS over RDMA). An option to increment the untagged receive credits attached to an untagged send would allow per Endpoint/QP credit tracking wihtout tying it to buffer replenishment or requiring any additional crossing of the system bus. There are reasons why the ULP is in a very poor position to enforce MSN validation: Messages with invalid MSNs must be accepted provisionally. Provisionally they consume resources, both buffers and CQEs. This may result in CQ overflow and/or buffer exhaustion *before* the ULP has a chance to respond. ULP diagnosis of MSN validity is asynchronous. The ULP cannot tell if an MSN that is valid by the time the ULP examines it had in fact "jumped the gun" and was sent before it had been authorized. If the ULP has decided to limit a client to 4 in-flight messages, it is important that they not be able to send 5. But it is next to impossible for the ULP to determine whether then nth request was sent before or after the n-5th reply had been sent. Having every client round up the number of in-flight messages allowed by one would have a serious impact on a multi-client server. In summary, underprovisioning is fine. It is even a good idea when done in a way that buffers shortfalls occur very rarely. ULPs that want to abdicate their responsibility to implement flow control are not fine. They cannot rely upon the local implementations to sweep such an abdication under the rug. Many non-RDMA ULPs rely upon the LLP to provide flow control for them. It is a beneficial side-effect of the undersirable bottleneck of traditional LLPs. RDMA eliminates that bottleneck. That means that ULPs will have to actually think about how they are flow controlled. Review the applicability draft and raise questions about the analysis there if you disagree. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 16 17:36:34 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA18650 for ; Wed, 16 Jul 2003 17:36:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ctwX-0007PI-HG for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 17:36:09 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6GLa93e028472 for rddp-archive@odin.ietf.org; Wed, 16 Jul 2003 17:36:09 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ctwX-0007P9-D7 for rddp-web-archive@optimus.ietf.org; Wed, 16 Jul 2003 17:36:09 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA18639 for ; Wed, 16 Jul 2003 17:36:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ctwV-0003lR-00 for rddp-web-archive@ietf.org; Wed, 16 Jul 2003 17:36:07 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ctwP-0003lM-00 for rddp-web-archive@ietf.org; Wed, 16 Jul 2003 17:36:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ctwO-0007NH-SQ; Wed, 16 Jul 2003 17:36:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ctwA-0007N0-SJ for rddp@optimus.ietf.org; Wed, 16 Jul 2003 17:35:47 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA18631 for ; Wed, 16 Jul 2003 17:35:41 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ctw8-0003lE-00 for rddp@ietf.org; Wed, 16 Jul 2003 17:35:44 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19ctvx-0003jv-00 for rddp@ietf.org; Wed, 16 Jul 2003 17:35:33 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 00947244965; Wed, 16 Jul 2003 17:32:19 -0400 (EDT) Date: Wed, 16 Jul 2003 16:32:10 -0500 From: Caitlin Bestler To: John Hufferd Cc: "Mallikarjun C." , pat_thaler@agilent.com, rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] Re: Untagged Message Validation Recap Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Wednesday, July 16, 2003, hufferd@us.ibm.com wrote: >Caitlin, >I do not think your characterization of the proposal is correct or fair. >No one has said what you are trying to read into our messages! Actually, I believe you are the one mistating your position. And the problem is that the mistatement creates a major problem. You want to be able to underprovision buffers. Fine. There is no disagreement with underprovisioning buffers, as long as it is done responsibly. But you are proposing doing so by *eliminating* the concept of an invalid or out-of-range MSN. That I object to. The minimal rules I proposed allow for under-provisioning of buffers. The only requirement is that the receiving side SHOULD stock an adequate supply, which I believe is totally consistent with the intent. I do not believe anyone is planning on severely under-provisioning the buffers. (Of course I would say that 95% confidence is *way* too low, but I don't think the IETF is the proper forum to debate how many nines are really required). The hard requirement that you objected to was that the data sink SHOULD reject an untagged message with an invalid MSN, and that it MUST reject an untagged message with an invalid MSN if accepting it would place reception of any untagged message with a valid MSN at risk. The criteria is whether or not the local ULP has authorized the reception -- *not* whether the local ULP has supplied a buffer. I do not expect a ULP like iSCSI over iWARP to pre-post a buffer for *every* possible untagged message. I do expect that the ULP will include some mechanism by which the Data Sink knows the maximum number of in-flight untagged messages, so that it can control whether or not those untagged messages will be recognized as valid or not. Even if you accept Pat's argument that it is unreasonable to require the DDP layer to accept this data, it would not mean that the data is not required. If you implemented using simple receive queues, that data would be needed. The ULP definition cannot *mandate* the use of under-provisioning. Foremost on the list of reasons is that an implementatin that was cleanly layered over the LLP has no choice but to terminate the stream when there are no available buffers. And we simply cannot require the Data Sink to provide buffer space for *invalid* messages. You are proposing a change to a protocol. Even if the intenet of your wording is not to do anything that would be abusive, if the language *allows* for implementations that will cause problems then there is a problem with the proposed language. And allowing an invalid untagged message to consume a buffer in *preference* to a valid untagged message is indeed a problem. It is a recipe for abusing network and server resources, as well as for denial of service attacks. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 17 05:30:34 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA26258 for ; Thu, 17 Jul 2003 05:30:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d55U-0001o2-QO for rddp-archive@odin.ietf.org; Thu, 17 Jul 2003 05:30:09 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6H9U8GE006935 for rddp-archive@odin.ietf.org; Thu, 17 Jul 2003 05:30:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d55U-0001nm-Kb for rddp-web-archive@optimus.ietf.org; Thu, 17 Jul 2003 05:30:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA26158 for ; Thu, 17 Jul 2003 05:30:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19d55R-0003tP-00 for rddp-web-archive@ietf.org; Thu, 17 Jul 2003 05:30:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19d55L-0003tM-00 for rddp-web-archive@ietf.org; Thu, 17 Jul 2003 05:29:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d55N-0001jx-KJ; Thu, 17 Jul 2003 05:30:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d55C-0001ic-CC for rddp@optimus.ietf.org; Thu, 17 Jul 2003 05:29:50 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id FAA26133 for ; Thu, 17 Jul 2003 05:29:45 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19d558-0003sh-00 for rddp@ietf.org; Thu, 17 Jul 2003 05:29:47 -0400 Received: from ztxmail04.ztx.compaq.com ([161.114.1.208]) by ietf-mx with esmtp (Exim 4.12) id 19d54j-0003lN-00 for rddp@ietf.org; Thu, 17 Jul 2003 05:29:21 -0400 Received: from cceexg13.americas.cpqcorp.net (cceexg13.americas.cpqcorp.net [16.110.250.119]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id 9DE07BCF2; Thu, 17 Jul 2003 04:23:09 -0500 (CDT) Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg13.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Thu, 17 Jul 2003 04:23:08 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [rddp] Some Minimal end-to-end Buffer Rules Required Date: Thu, 17 Jul 2003 04:23:08 -0500 Message-ID: <4D027986353D1341ADA4F2A4F8A170762AB1E765@cceexc18.americas.cpqcorp.net> Thread-Topic: [rddp] Some Minimal end-to-end Buffer Rules Required Thread-Index: AcNLpCMN9AEDJxpkTsCOJwD11Q1g8AAnJXeg From: "Culley, Paul" To: "Caitlin Bestler" , "John Hufferd" Cc: X-OriginalArrivalTime: 17 Jul 2003 09:23:08.0759 (UTC) FILETIME=[09377670:01C34C45] Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com]=20 > Sent: Wednesday, July 16, 2003 9:10 AM > To: John Hufferd > Cc: rddp@ietf.org > Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required >=20 >=20 Deleted... >=20 > Then these applications do not work over InfiniBand? You have > a credit limit over IB RC, and it is even less flexible than=20 > what you can do with iWARP. >=20 Actually these applications (that occasionally send more messages than normally credited) could work quite well on IB because IB supports automatic flow control for the RC mode receive queue buffers. In the "normal" case, the flow control is not invoked, everything flows smoothly. In the exceptional case when too many messages need to be sent, IB backs up the messages at the sender (in the Send Queue). Paul C. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 17 09:41:34 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA06145 for ; Thu, 17 Jul 2003 09:41:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d90O-0001zj-8w for rddp-archive@odin.ietf.org; Thu, 17 Jul 2003 09:41:08 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6HDf8m3007663 for rddp-archive@odin.ietf.org; Thu, 17 Jul 2003 09:41:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d90O-0001zW-0g for rddp-web-archive@optimus.ietf.org; Thu, 17 Jul 2003 09:41:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA06132 for ; Thu, 17 Jul 2003 09:41:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19d90M-0006XL-00 for rddp-web-archive@ietf.org; Thu, 17 Jul 2003 09:41:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19d90G-0006XI-00 for rddp-web-archive@ietf.org; Thu, 17 Jul 2003 09:41:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d90H-0001yr-DI; Thu, 17 Jul 2003 09:41:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d8zX-0001yR-RR for rddp@optimus.ietf.org; Thu, 17 Jul 2003 09:40:15 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA06117 for ; Thu, 17 Jul 2003 09:40:11 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19d8zW-0006Wx-00 for rddp@ietf.org; Thu, 17 Jul 2003 09:40:14 -0400 Received: from palrel12.hp.com ([156.153.255.237]) by ietf-mx with esmtp (Exim 4.12) id 19d8zK-0006Wq-00 for rddp@ietf.org; Thu, 17 Jul 2003 09:40:03 -0400 Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel12.hp.com (Postfix) with ESMTP id 4E2A51C00F66 for ; Thu, 17 Jul 2003 06:39:42 -0700 (PDT) Received: from mk731915.cup.hp.com ([15.244.168.187]) by esmail.cup.hp.com (8.11.1/8.8.6) with ESMTP id h6HDYJa27479 for ; Thu, 17 Jul 2003 06:34:20 -0700 (PDT) Message-Id: <5.1.0.14.2.20030717055557.021e0c10@15.13.130.73> X-Sender: krause@15.13.130.73 X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Thu, 17 Jul 2003 06:00:45 -0700 To: rddp@ietf.org From: Michael Krause Subject: RE: [rddp] Some Minimal end-to-end Buffer Rules Required In-Reply-To: <4D027986353D1341ADA4F2A4F8A170762AB1E765@cceexc18.americas .cpqcorp.net> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 04:23 AM 7/17/2003 -0500, Culley, Paul wrote: > > -----Original Message----- > > From: Caitlin Bestler [mailto:cait@asomi.com] > > Sent: Wednesday, July 16, 2003 9:10 AM > > To: John Hufferd > > Cc: rddp@ietf.org > > Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required > > > > >Deleted... > > > > Then these applications do not work over InfiniBand? You have > > a credit limit over IB RC, and it is even less flexible than > > what you can do with iWARP. > > >Actually these applications (that occasionally send more messages than >normally credited) could work quite well on IB because IB supports >automatic flow control for the RC mode receive queue buffers. In the >"normal" case, the flow control is not invoked, everything flows >smoothly. In the exceptional case when too many messages need to be >sent, IB backs up the messages at the sender (in the Send Queue). Exactly. IB was designed with a send credit scheme that would allow an application to post as many Send operations as it desired (up to its SQ limit) knowing the that underlying interconnect would insure that no more than what had been actually credited by the receiver would be sent at any given time. A simple example: application has a lot of work to do, gets a time slice, posts a lot of work, completes it time slice. It does not know if the remote connection is resource thin or rich thus it relies on the credit scheme to allow the remote to advertise its credit as a function of its resource capabilities for this particular connection. Please note that all of this is focused on data center distances which many viewed as to why it was a practical idea; consensus on this list was the credits were not good due to supporting WAN distances, analysis of existing ULP which already use a credit scheme at the ULP level, etc. Mike _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 17 09:53:37 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA06540 for ; Thu, 17 Jul 2003 09:53:37 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9C0-0002Se-2d for rddp-archive@odin.ietf.org; Thu, 17 Jul 2003 09:53:08 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6HDr8jK009454 for rddp-archive@odin.ietf.org; Thu, 17 Jul 2003 09:53:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9Bz-0002SP-FC for rddp-web-archive@optimus.ietf.org; Thu, 17 Jul 2003 09:53:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA06527 for ; Thu, 17 Jul 2003 09:53:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19d9Bx-0006aE-00 for rddp-web-archive@ietf.org; Thu, 17 Jul 2003 09:53:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19d9Br-0006aB-00 for rddp-web-archive@ietf.org; Thu, 17 Jul 2003 09:52:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9Bs-0002Qj-HP; Thu, 17 Jul 2003 09:53:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9B4-0002Q1-RF for rddp@optimus.ietf.org; Thu, 17 Jul 2003 09:52:10 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA06491 for ; Thu, 17 Jul 2003 09:52:06 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19d9B2-0006a7-00 for rddp@ietf.org; Thu, 17 Jul 2003 09:52:08 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19d9As-0006a2-00 for rddp@ietf.org; Thu, 17 Jul 2003 09:51:58 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 364E0245BDC; Thu, 17 Jul 2003 09:51:12 -0400 (EDT) Date: Thu, 17 Jul 2003 08:51:03 -0500 From: Caitlin Bestler Subject: RE: [rddp] Some Minimal end-to-end Buffer Rules Required To: "Culley, Paul" Cc: John Hufferd , rddp@ietf.org X-Priority: 3 In-Reply-To: <4D027986353D1341ADA4F2A4F8A170762AB1E765@cceexc18.americas.cpqcorp.net> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Thursday, July 17, 2003, paul.culley@hp.com wrote: >> Then these applications do not work over InfiniBand? You have >> a credit limit over IB RC, and it is even less flexible than >> what you can do with iWARP. >> >Actually these applications (that occasionally send more messages than >normally credited) could work quite well on IB because IB supports >automatic flow control for the RC mode receive queue buffers. In the >"normal" case, the flow control is not invoked, everything flows >smoothly. In the exceptional case when too many messages need to be >sent, IB backs up the messages at the sender (in the Send Queue). Correct. But that service is not provided by the RDMA layer with iWARP because it would require an iWARP layer ack message. A minimal adaptation would be for the application to queue them itself, and release them in response to ULP acks (associated with untagged messages). I am assuming that there is still agreement that while transport layer queuing is a nice feature, it does not justify additional wire traffic. Agreed? _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 17 10:13:33 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA08222 for ; Thu, 17 Jul 2003 10:13:33 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9VL-0003Y0-JP for rddp-archive@odin.ietf.org; Thu, 17 Jul 2003 10:13:07 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6HED7cR013630 for rddp-archive@odin.ietf.org; Thu, 17 Jul 2003 10:13:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9VL-0003Xl-GL for rddp-web-archive@optimus.ietf.org; Thu, 17 Jul 2003 10:13:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA08158 for ; Thu, 17 Jul 2003 10:13:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19d9VJ-0006i1-00 for rddp-web-archive@ietf.org; Thu, 17 Jul 2003 10:13:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19d9VD-0006hy-00 for rddp-web-archive@ietf.org; Thu, 17 Jul 2003 10:12:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9VF-0003WE-1o; Thu, 17 Jul 2003 10:13:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9Uh-0003Vq-Iv for rddp@optimus.ietf.org; Thu, 17 Jul 2003 10:12:27 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA08082 for ; Thu, 17 Jul 2003 10:12:22 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19d9Uf-0006hJ-00 for rddp@ietf.org; Thu, 17 Jul 2003 10:12:25 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19d9UU-0006gf-00 for rddp@ietf.org; Thu, 17 Jul 2003 10:12:14 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 21D6924488E; Thu, 17 Jul 2003 10:11:45 -0400 (EDT) Date: Thu, 17 Jul 2003 09:11:36 -0500 From: Caitlin Bestler Subject: RE: [rddp] Some Minimal end-to-end Buffer Rules Required To: Michael Krause Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <5.1.0.14.2.20030717055557.021e0c10@15.13.130.73> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Thursday, July 17, 2003, krause@cup.hp.com wrote: > >Exactly. IB was designed with a send credit scheme that would >allow an application to post as many Send operations as it >desired (up to its SQ limit) knowing the that underlying >interconnect would insure that no more than what had been >actually credited by the receiver would be sent at any given >time. A simple example: application has a lot of work to do, >gets a time slice, posts a lot of work, completes it time >slice. It does not know if the remote connection is resource >thin or rich thus it relies on the credit scheme to allow the >remote to advertise its credit as a function of its resource >capabilities for this particular connection. Please note that >all of this is focused on data center distances which many >viewed as to why it was a practical idea; consensus on this >list was the credits were not good due to supporting WAN >distances, analysis of existing ULP which already use a credit >scheme at the ULP level, etc. > Two quibbles: 1) This discussion actually occurred during the yahoogroups era, so it technically wasn't "this list". 2) The WAN implications were not merely distance, but the possibility of intermediate termination of the TCP stream. Renato's early drafts for iWARP had IB-like send credits, but they relied upon the LLP ack as feedback. TCP middleboxes are allowed to terminate and re-originate a stream, making a TCP ack meaningless to the iWARP sender. Given that limitation, it was agreed that adding an explicit iWARP ack was not a viable solution. Therefore the credits had to be deleted, which was reflected in the first IETF drafts. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 17 10:32:34 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA09351 for ; Thu, 17 Jul 2003 10:32:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9nk-0004Xc-Hs for rddp-archive@odin.ietf.org; Thu, 17 Jul 2003 10:32:09 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6HEW8pW017455 for rddp-archive@odin.ietf.org; Thu, 17 Jul 2003 10:32:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9nk-0004XS-Ef for rddp-web-archive@optimus.ietf.org; Thu, 17 Jul 2003 10:32:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA09312 for ; Thu, 17 Jul 2003 10:32:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19d9ni-0006pn-00 for rddp-web-archive@ietf.org; Thu, 17 Jul 2003 10:32:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19d9nc-0006pk-00 for rddp-web-archive@ietf.org; Thu, 17 Jul 2003 10:32:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9ne-0004Vn-0c; Thu, 17 Jul 2003 10:32:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9mq-0004Tq-MC for rddp@optimus.ietf.org; Thu, 17 Jul 2003 10:31:12 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA09234 for ; Thu, 17 Jul 2003 10:31:07 -0400 (EDT) From: pat_thaler@agilent.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19d9mo-0006oW-00 for rddp@ietf.org; Thu, 17 Jul 2003 10:31:10 -0400 Received: from msgbas1x.cos.agilent.com ([192.25.240.36]) by ietf-mx with esmtp (Exim 4.12) id 19d9md-0006o9-00 for rddp@ietf.org; Thu, 17 Jul 2003 10:30:59 -0400 Received: from relcos1.cos.agilent.com (relcos1.cos.agilent.com [130.29.152.239]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id 4BF8816A8C; Thu, 17 Jul 2003 08:30:17 -0600 (MDT) Received: from axcsbh3.cos.agilent.com (axcsbh3.cos.agilent.com [130.29.152.190]) by relcos1.cos.agilent.com (Postfix) with SMTP id 7A8C62A8; Thu, 17 Jul 2003 08:30:19 -0600 (MDT) Received: from 130.29.152.190 by axcsbh3.cos.agilent.com (InterScan E-Mail VirusWall NT); Thu, 17 Jul 2003 08:30:04 -0600 Received: by axcsbh3.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Thu, 17 Jul 2003 08:30:04 -0600 Message-ID: <1BEBA5E8600DD4119A50009027AF54A013856AC5@axcs04.cos.agilent.com> To: paul.culley@hp.com, cait@asomi.com, hufferd@us.ibm.com Cc: rddp@ietf.org Subject: RE: [rddp] Some Minimal end-to-end Buffer Rules Required Date: Thu, 17 Jul 2003 08:30:02 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , In addition to Paul's note, it is also important to realize that the needs of any application working efficiently over the internet are different than the needs when working over Infiniband. Infiniband networks can generally be counted on to be small and are built with devices designed with the very low latency desired for IPC interactions. Round trip delays are therefore tiny compared to those on the Internet. Therefore, on an Infiniband network the round trip delays in requesting and receiving additional credits will not significantly affect performance. That can not be said for systems operating over internets. Pat -----Original Message----- From: Culley, Paul [mailto:paul.culley@hp.com] Sent: Thursday, July 17, 2003 2:23 AM To: Caitlin Bestler; John Hufferd Cc: rddp@ietf.org Subject: RE: [rddp] Some Minimal end-to-end Buffer Rules Required > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Wednesday, July 16, 2003 9:10 AM > To: John Hufferd > Cc: rddp@ietf.org > Subject: Re: [rddp] Some Minimal end-to-end Buffer Rules Required > > Deleted... > > Then these applications do not work over InfiniBand? You have > a credit limit over IB RC, and it is even less flexible than > what you can do with iWARP. > Actually these applications (that occasionally send more messages than normally credited) could work quite well on IB because IB supports automatic flow control for the RC mode receive queue buffers. In the "normal" case, the flow control is not invoked, everything flows smoothly. In the exceptional case when too many messages need to be sent, IB backs up the messages at the sender (in the Send Queue). Paul C. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 17 10:43:35 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA09610 for ; Thu, 17 Jul 2003 10:43:35 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9yQ-0005DS-J9 for rddp-archive@odin.ietf.org; Thu, 17 Jul 2003 10:43:10 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6HEhA5S020044 for rddp-archive@odin.ietf.org; Thu, 17 Jul 2003 10:43:10 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9yQ-0005DD-Bg for rddp-web-archive@optimus.ietf.org; Thu, 17 Jul 2003 10:43:10 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA09597 for ; Thu, 17 Jul 2003 10:43:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19d9yN-0006ro-00 for rddp-web-archive@ietf.org; Thu, 17 Jul 2003 10:43:08 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19d9yI-0006rl-00 for rddp-web-archive@ietf.org; Thu, 17 Jul 2003 10:43:02 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9yG-000579-Io; Thu, 17 Jul 2003 10:43:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19d9y3-00056g-Um for rddp@optimus.ietf.org; Thu, 17 Jul 2003 10:42:47 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA09588 for ; Thu, 17 Jul 2003 10:42:39 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19d9xy-0006rd-00 for rddp@ietf.org; Thu, 17 Jul 2003 10:42:42 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19d9xn-0006ra-00 for rddp@ietf.org; Thu, 17 Jul 2003 10:42:32 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 478D1244E50; Thu, 17 Jul 2003 10:42:00 -0400 (EDT) Date: Thu, 17 Jul 2003 09:41:52 -0500 From: Caitlin Bestler Subject: RE: [rddp] Some Minimal end-to-end Buffer Rules Required To: pat_thaler@agilent.com, rddp@ietf.org X-Priority: 3 In-Reply-To: <1BEBA5E8600DD4119A50009027AF54A013856AC5@axcs04.cos.agilent.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Thursday, July 17, 2003, pat_thaler@agilent.com wrote: >In addition to Paul's note, it is also important to realize that the >needs of any application working efficiently over the internet are >different than the needs when working over Infiniband. Infiniband >networks can generally be counted on to be small and are built with >devices designed with the very low latency desired for IPC interactions. >Round trip delays are therefore tiny compared to those on the Internet. >Therefore, on an Infiniband network the round trip delays in requesting >and receiving additional credits will not significantly affect >performance. That can not be said for systems operating over internets. Good point, not only would an extra message be required, it would be slightly flaky in its effectiveness as well. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Jul 18 15:03:42 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08812 for ; Fri, 18 Jul 2003 15:03:41 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19daVf-0001DU-4g for rddp-archive@odin.ietf.org; Fri, 18 Jul 2003 15:03:15 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6IJ3FNo004672 for rddp-archive@odin.ietf.org; Fri, 18 Jul 2003 15:03:15 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19daVf-0001DH-1X for rddp-web-archive@optimus.ietf.org; Fri, 18 Jul 2003 15:03:15 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08755 for ; Fri, 18 Jul 2003 15:03:11 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19daVc-0002l7-00 for rddp-web-archive@ietf.org; Fri, 18 Jul 2003 15:03:12 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19daVV-0002kr-00 for rddp-web-archive@ietf.org; Fri, 18 Jul 2003 15:03:05 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19daVR-0001Ba-JP; Fri, 18 Jul 2003 15:03:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19daVM-0001BF-IW for rddp@optimus.ietf.org; Fri, 18 Jul 2003 15:02:56 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08741 for ; Fri, 18 Jul 2003 15:02:52 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19daVJ-0002kl-00 for rddp@ietf.org; Fri, 18 Jul 2003 15:02:53 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19daV8-0002kB-00 for rddp@ietf.org; Fri, 18 Jul 2003 15:02:42 -0400 Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 7F653245491 for ; Fri, 18 Jul 2003 15:01:48 -0400 (EDT) Date: Fri, 18 Jul 2003 14:01:40 -0500 From: Caitlin Bestler To: rddp@ietf.org X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 2.0 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] Security Implications of sharing buffers/credits Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit I've been working through my object models to determine exactly what sorts of sharing related to untagged messages can be supported, and the implications as to performance/implementation/interoperability/security. The performance/implementation implications I'll be keeping to myself, but I thought I'd share the interoperability/security notes I generated. There are two critical issues regarding reception of untagged messages: -- How the MSN is validated. -- When a buffer is associated with an untagged message. These issues will be dealt with on a per-strategy basis. Strategy #1: Simple Ordered Receive Queue This is the classic RDMA interface. Buffers are pre-posted on a per stream basis by the ULP. As each is posted it is pre-assigned to the next MSN. Validity of an MSN is determined by whether or not there is already an associated buffer when a segment with the MSN is first received. Strategy #2: Shared Buffers / Implicit Shared Credits with soft per-stream limits. This is the Shared Receive Queue strategy. Buffers are posted to a Shared Buffer Pool (it is not really a queue). Credits are not tracked on a per-stream basis. When a DDP Segment for an MSN is first received, it is presumed valid. A buffer is allocated from the pool. The number of buffers allocated can be checked against a soft limit. This allows detection of "out-of-control" streams. However this is a soft limit, because any response is left to the ULP. There is an important corollary issue: what the verbs draft refers to as sequential-ordering versus arrival-ordering. With sequential-ordering, allocation of a buffer for MSN x requires allocation of a buffer for all previous MSNs. This allows a single message to tie up a large number of buffers. While this will undoubtedly trigger a soft-limit alarm, those buffers are tied up until the ULP acts. With arrival-ordering, the arrival of any new MSN can consume at most one buffer. However, this can lead to a state where the itnermediate MSNs cannot be received because all available buffers are being held by later MSNs. The later messages cannot be delivered, therefore the buffers cannot be released. With SAcks these messages may not be retroactively releasable. This is particularly relevant when the LLP is SCTP because TSN ordering is not necessarily related to DDP sequencing. The SCTP stack is free to re-order to optimize chunk bundling. Therefore having acked a non-deliverable untagged message is a distinct possibility. With the availability of the soft-limit ordering, the maximum damage from an abusive stream can be made predictable to the extent that the ULP's response time to an asynchronous error can be made predictable. In many environments this will require a substantial increase in the allocation of "spare" resources. Care must be taken in scoping and sizing the Completion Queue when this strategy is employed. - The CQ may be sized to allow for every buffer in the pool to be completed to it. This will generally be wasteful unless the CQ has the same scope as the pool. - The CQ may be scoped to a single DDP Stream. Excessive completions triggered by the remote peer will be incapable of damaging any other DDP Stream. Strategy #3: Shared Buffers / Explicit Per-stream Credits This is the Buffer Pool strategy proposed in prior emails. Buffers posted are placed in a potentially shared Buffer Pool, with a credit being issued to the specific DDP stream. Each credit pre-enables (or validates) the next MSN. Unlike the Simple Ordered Receive Queue model, the MSN is validated *without* pre-assigning a buffer to it. The ULP may also grant credits *without* supplying a buffer, thereby underprovisioning the pool. When a DDP Segment for an MSN is first received it is considered valid if the MSN is within the enabled range for that stream. At that time a buffer must be associated. The absence of a buffer for a valid MSN is a fault. Recovery procedures are a local issue, but ultimately must include termination of the stream. As with any under-provisioning strategy, there is a chance of running dry. However unauthorized actions by the remote peer cannot provoke such a shortage. This strategy is consistent with bother Arrival Ordering and Sequential Ordering as discussed above. However, this is less of an issue with this strategy. Sequential ordering can only allocate buffers for messages that are authorized anyway. Stranded undeliverable messages can only occur if there is a severe under-allocation of buffers because the stranded message had a pre-validated MSN anyway. The existence of hard per-stream limits on the number of valid MSNs also means that the number of completions is predicatable on a per stream basis. This allows safe pooling of completion queues across multiple DDP Streams without having to match the scope of the Buffer Pool. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 21 13:46:55 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05079 for ; Mon, 21 Jul 2003 13:46:55 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19eek1-0006Uy-18 for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 13:46:29 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6LHkTBc024976 for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 13:46:29 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19eejz-0006Uk-HM for rddp-web-archive@optimus.ietf.org; Mon, 21 Jul 2003 13:46:27 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05056 for ; Mon, 21 Jul 2003 13:46:23 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19eejx-0004wK-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 13:46:25 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19eejr-0004w7-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 13:46:19 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19eejZ-0006Qy-Bd; Mon, 21 Jul 2003 13:46:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19eeir-0006QS-QJ for rddp@optimus.ietf.org; Mon, 21 Jul 2003 13:45:17 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05008 for ; Mon, 21 Jul 2003 13:45:13 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19eeip-0004vd-00 for rddp@ietf.org; Mon, 21 Jul 2003 13:45:15 -0400 Received: from mail4.microsoft.com ([131.107.3.122]) by ietf-mx with esmtp (Exim 4.12) id 19eeie-0004v4-00 for rddp@ietf.org; Mon, 21 Jul 2003 13:45:04 -0400 Received: from inet-vrs-04.redmond.corp.microsoft.com ([157.54.8.149]) by mail4.microsoft.com with Microsoft SMTPSVC(5.0.2195.6713); Mon, 21 Jul 2003 10:44:07 -0700 Received: from 157.54.8.109 by inet-vrs-04.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Mon, 21 Jul 2003 10:44:07 -0700 Received: from red-imc-01.redmond.corp.microsoft.com ([157.54.9.102]) by inet-hub-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Mon, 21 Jul 2003 10:44:10 -0700 Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-01.redmond.corp.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Mon, 21 Jul 2003 10:44:04 -0700 Received: from WIN-MSG-10.wingroup.windeploy.ntdev.microsoft.com ([157.54.12.81]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Mon, 21 Jul 2003 10:43:57 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------InterScan_NT_MIME_Boundary" Date: Mon, 21 Jul 2003 10:44:11 -0700 Message-ID: Thread-Topic: security doc - Definition of Trust Thread-Index: AcNNQP+0kJdXtq5PQqyuuaxDi/CNGw== From: "Jim Pinkerton" To: X-OriginalArrivalTime: 21 Jul 2003 17:43:57.0755 (UTC) FILETIME=[A97760B0:01C34FAF] Subject: [rddp] security doc - Definition of Trust Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multi-part message in MIME format. --------------InterScan_NT_MIME_Boundary Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C34FAF.B1B284BA" ------_=_NextPart_001_01C34FAF.B1B284BA Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable =20 One of the issues that has come up on the reflector is that the definition of Trust should be changed to "Partial Trust", with an appropriate change to the definition. The existing draft defines trust as: =20 Trust - When one party depends upon the other party to not subvert the goals of the protocols, e.g., it will not attempt to perform the following attacks: spoofing, repudiation, information disclosure, denial of service, or elevation of privilege.=20 An Untrusted peer is a party that may (or may not) attempt to perform one or more of the above attacks. A partially trusted peer (either the Local Peer or Remote Peer) may be trusted to not attempt to perform some subset of the above attacks, but not trusted to perform a different subset. Possible new definition (feedback/wordsmithing appreciated - I want general agreement on this before I go through the spec): =20 Partial Trust - When one party depends upon the other party to not attempt to perform a specific attack or set of attacks. A partially trusted peer (either the Local Peer or Remote Peer) may be trusted to not attempt to perform any subset of the specific list of attacks. However, there is no guarantee that the Peer won't try to perform a different set of attacks. An Untrusted peer is a party that may (or may not) attempt to perform one or more of the listed set of attacks.=20 Jim =20 =20 =20 =20 ------_=_NextPart_001_01C34FAF.B1B284BA Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

 

One of the issues that has come up on the reflector = is that the definition of Trust should be changed to “Partial = Trust”, with an appropriate change to the definition. The existing draft defines = trust as:

 

Trust - When one party depends upon the other = party to not subvert the goals of the protocols, e.g., it will not attempt to = perform the following attacks: spoofing, repudiation, information disclosure, = denial of service, or elevation of privilege.

An Untrusted peer is a party that may (or may = not) attempt to perform one or more of the above attacks. A partially trusted = peer (either the Local Peer or Remote Peer) may be trusted to not attempt to = perform some subset of the above attacks, but not trusted to perform a different subset.

Possible new definition (feedback/wordsmithing = appreciated – I want general agreement on this before I go through the = spec):

 

Partial Trust - When one party depends upon = the other party to not attempt to perform a specific attack or set of attacks. A partially trusted peer (either the Local Peer or Remote Peer) may be = trusted to not attempt to perform any subset of the specific list of attacks. = However, there is no guarantee that the Peer won’t try to perform a = different set of attacks.

An Untrusted peer is a party that may (or may = not) attempt to perform one or more of the listed set of attacks. =

Jim

 

 

 

 

------_=_NextPart_001_01C34FAF.B1B284BA-- --------------InterScan_NT_MIME_Boundary-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 21 14:16:33 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA05670 for ; Mon, 21 Jul 2003 14:16:33 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19efCh-0007QJ-D3 for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 14:16:07 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6LIG7Si028535 for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 14:16:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19efCh-0007QA-6z for rddp-web-archive@optimus.ietf.org; Mon, 21 Jul 2003 14:16:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA05664 for ; Mon, 21 Jul 2003 14:16:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19efCe-00058z-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 14:16:04 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19efCY-00058w-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 14:15:58 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19efCa-0007OK-8Y; Mon, 21 Jul 2003 14:16:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ef89-0007LR-0C for rddp@optimus.ietf.org; Mon, 21 Jul 2003 14:11:25 -0400 Received: from smtpout.mac.com (A17-250-248-89.apple.com [17.250.248.89]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA05616 for ; Mon, 21 Jul 2003 14:11:18 -0400 (EDT) Received: from mac.com (smtpin08-en2 [10.13.10.153]) by smtpout.mac.com (Xserve/MantshX 2.0) with ESMTP id h6LIB17r005414; Mon, 21 Jul 2003 11:11:01 -0700 (PDT) Received: from mac.com (64-144-5-25.client.dsl.net [64.144.5.25]) (authenticated bits=0) by mac.com (Xserve/8.12.9/MantshX 2.0) with ESMTP id h6LIAvNr026229; Mon, 21 Jul 2003 11:10:58 -0700 (PDT) Date: Mon, 21 Jul 2003 13:10:55 -0500 Subject: Re: [rddp] security doc - Definition of Trust Content-Type: text/plain; charset=WINDOWS-1252; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: To: "Jim Pinkerton" From: Caitlin Bester In-Reply-To: Message-Id: Content-Transfer-Encoding: quoted-printable X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable On Monday, July 21, 2003, at 12:44 PM, Jim Pinkerton wrote: > =A0 > > One of the issues that has come up on the reflector is that the=20 > definition of Trust should be changed to =93Partial Trust=94, with an=20= > appropriate change to the definition. The existing draft defines trust=20= > as: > > =A0 > > Trust - When one party depends upon the other party to not subvert the=20= > goals of the protocols, e.g., it will not attempt to perform the=20 > following attacks: spoofing, repudiation, information disclosure,=20 > denial of service, or elevation of privilege. > > An Untrusted peer is a party that may (or may not) attempt to perform=20= > one or more of the above attacks. A partially trusted peer (either the=20= > Local Peer or Remote Peer) may be trusted to not attempt to perform=20 > some subset of the above attacks, but not trusted to perform a=20 > different subset. > > Possible new definition (feedback/wordsmithing appreciated =96 I want=20= > general agreement on this before I go through the spec): > > =A0 > > Partial Trust - When one party depends upon the other party to not=20 > attempt to perform a specific attack or set of attacks. A partially=20 > trusted peer (either the Local Peer or Remote Peer) may be trusted to=20= > not attempt to perform any subset of the specific list of attacks.=20 > However, there is no guarantee that the Peer won=92t try to perform a=20= > different set of attacks. > > An Untrusted peer is a party that may (or may not) attempt to perform=20= > one or more of the listed set of attacks. > > Jim > I believe there needs to be a distinction between "trust not to launch=20= an explicit attack" and "trust to conform to the ULP". To a large degree, the intent of the peer is irrelevant. The question=20 is really relates to how paranoid must an application be under what circumstances. For example, while it will always be possible for the ULP to tailor=20 tagged buffer exposure to *exactly* the degree required, a local ULP could easily find this inconvenient to do. If may elect to=20= enable an STag covering a set of buffers, and rely upon its peer to only write to the portion that was enabled per the ULP. However, it should not do so if it imperils any other process or any=20 other party by doing so. If you use a shared buffer pool for untagged messages, and the scope of=20= that sharing encompasses DDP Streams with multiple clients, then your trust of Client A should not place your=20 interactions with Client B at risk (I would almost be tempted to say MUST NOT). Here is a very specific example where Client A may not be "attacking".=20= Client A may merely be a bit over-eager and sending the nTh request before receving the n-mth response and thereby=20= has more than m requests pending. These "cheating clients" have no intention of bringing the server down,=20= but if there are enough of them they will drain the shared buffer pool and eventually force another client's=20 stream to be terminated (or possibly only undergo a false drop). At that point they have damaged the other=20 client, but there was never an intention to "attack". Now just to be safe, it is probably worth stating that there are a set=20= of known "attacks" which are automatically considered to be "non-compliant" with the parent ULP. At least by=20 default, the ULP could be something like a hacker contest where you are *encouraged* to forge your address. But=20= otherwise I think we are quite safe stating that the type of attacks explicitly listed so far are=20 presumed to be non-compliant with the ULP. But the ultimate goal should be to make it clear that: -- The local interface MUST provide tools to allow the ULP to protect=20 itself. -- The local ULP MUST use these tools to enable its peer to perform as=20= specified under the Upper Layer Protocol, -- The local ULP MAY enable more access than what is required to=20 simplify its use of the local interface, but SHOULD NOT do so if it imperils any other process or any other=20 sessions. But the need to protect yourself from stray bullets exists whether the=20= shooter was aiming for you or not. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 21 15:15:35 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08171 for ; Mon, 21 Jul 2003 15:15:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19eg7n-0001n8-Uo for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 15:15:08 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6LJF7qX006883 for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 15:15:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19eg7n-0001mw-PJ for rddp-web-archive@optimus.ietf.org; Mon, 21 Jul 2003 15:15:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08123 for ; Mon, 21 Jul 2003 15:15:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19eg7m-0005bn-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 15:15:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19eg7h-0005bk-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 15:15:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19eg7g-0001mS-Ns; Mon, 21 Jul 2003 15:15:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19eg71-0001m2-76 for rddp@optimus.ietf.org; Mon, 21 Jul 2003 15:14:19 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08048 for ; Mon, 21 Jul 2003 15:14:15 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19eg6z-0005b7-00 for rddp@ietf.org; Mon, 21 Jul 2003 15:14:17 -0400 Received: from mxic2.corp.emc.com ([128.221.31.40]) by ietf-mx with esmtp (Exim 4.12) id 19eg6p-0005aH-00 for rddp@ietf.org; Mon, 21 Jul 2003 15:14:07 -0400 Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Mon, 21 Jul 2003 15:12:57 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0C246750@corpmx14.us.dg.com> To: rddp@ietf.org Date: Mon, 21 Jul 2003 15:02:43 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] WG chair on buffer exhaustion Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , The heated debate on whether lack of availability of a buffer for an untagged message causes immediate termination of the DDP stream on which it arrived caused some additional email behind the scenes in addition to what's been on the list. With my WG chair hat on, here is what I believe to be the correct resolution of this issue: The request to do something other than immediately terminate a stream when a message shows up that does not have a pre-queued untagged buffer to receive it does not require that DDP tell TCP not to ACK the message; an implementation could choose to do that based on inter-layer implementation optimizations, but had better do so very rarely. Alternatively, a DDP implementation could choose to have some "hidden" resources in its back pocket that might involve a less efficient receive path (e.g., receive the errant message to a hidden kernel buffer and then copy to an application buffer when it becomes available) - again, this would need to be a rare occurrence for obvious reasons. This sort of approach is more robust than having TCP not ACK, as it avoids propagating a host problem into the network and puts the performance penalty for the implementation problem (should have had a buffer available, but didn't) squarely on the system that had the problem. I would prefer to see the "hidden [kernel] buffer" approach described as an example rather than the "TCP doesn't ACK" approach. Among the other problems with "TCP doesn't ACK" is that it creates a nasty head-of-line blocking problem that affects all traffic on the connection, not just the untagged traffic that is facing the resource shortage. OTOH, if the ULP has set a hard limit on the number of untagged messages that can come in from a specific stream, and that stream exceeds the limit, this is evidence that the counterparty isn't following the rules, and termination of the stream is a reasonable response. In the particular scenario that John raises below, the hard limit would need to be set high enough to accommodate error/special/ emergency conditions, and statistical/shared buffer provisioning would probably be used to avoid having to provision buffers for simultaneous occurrence of error/special/emergency conditions on all inbound streams. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- > > > >On Wednesday, July 16, 2003, hufferd@us.ibm.com wrote: > > > >>Again, I think that Mallikarjun stated things correctly. There are > >>important applications which can only correctly predict the number of > >>buffers needed in non error/special/emergency conditions. When > >>error/special/emergency conditions occur, the number and urgency of the > >>handling is problematical. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 21 15:28:34 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08554 for ; Mon, 21 Jul 2003 15:28:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19egKN-0002Ch-To for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 15:28:07 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6LJS745008470 for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 15:28:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19egKN-0002CX-Pg for rddp-web-archive@optimus.ietf.org; Mon, 21 Jul 2003 15:28:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08537 for ; Mon, 21 Jul 2003 15:28:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19egKM-0005iX-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 15:28:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19egKG-0005iU-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 15:28:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19egKG-0002Aq-K0; Mon, 21 Jul 2003 15:28:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19egJf-0002AJ-Cl for rddp@optimus.ietf.org; Mon, 21 Jul 2003 15:27:23 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08516 for ; Mon, 21 Jul 2003 15:27:19 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19egJe-0005i2-00 for rddp@ietf.org; Mon, 21 Jul 2003 15:27:22 -0400 Received: from maho3msx2.corp.emc.com ([128.221.11.32]) by ietf-mx with esmtp (Exim 4.12) id 19egJT-0005hj-00 for rddp@ietf.org; Mon, 21 Jul 2003 15:27:11 -0400 Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <3W9QWN3N>; Mon, 21 Jul 2003 15:26:11 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0C246751@corpmx14.us.dg.com> To: rddp@ietf.org Date: Mon, 21 Jul 2003 15:15:58 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] CRC rough consensus from Vienna Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , I'm just catching up on sending what happened in Vienna to the list. Sorry for the delay, but a significant piece of the explanation is at: http://money.cnn.com/2003/07/21/news/international/ba_strike.reut/ although they left out the part about thousands of pieces of luggage (including mine) going astray - my wife and I spent an unexpected night at a Heathrow airport motel without our luggage. Nonetheless, here's what I believe is the WG rough consensus to the last (I hope!) open CRC issue - one vs. two CRCs for the TCP mapping of the RDDP protocols. For anyone keeping track, this issue is hereby tagged as CRC issue (B1). After discussion in Vienna, I believe the WG rough consensus is for a single CRC rather than separate header and data CRCs. The discussion that is the basis for this rough consensus acknowledges that there are some advantages to the split CRCs, principally that the header CRC can be verified without waiting for the data, especially if the data spans more than one packet. The principal downside is increased complexity of two CRCs. The advantages of two CRCs are not overwhelming, as for aligned traffic, the data and its CRC will arrive immediately, and in the unaligned case, an implementation that wants to use the header for immediate placement without waiting for a second packet will be able to verify the TCP checksum before placement. It is also ok in an error case to tell the ULP that "something went wrong, the data in the buffer you handed to RDDP for placement is invalid as it may have been overwritten in part" - the TCP connection on which this happened will generally be terminated in this case. This does not appear to rule out or unfairly disadvantage any important classes of implementations. My recollection of past discussion is that I need to call this rough consensus over Jim Williams's objections, but comments from Jim and anyone else who takes issue with this position should be sent to the list. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 21 15:35:34 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08602 for ; Mon, 21 Jul 2003 15:35:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19egRA-0002RM-2J for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 15:35:08 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6LJZ8K3009376 for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 15:35:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19egR9-0002R9-Uv for rddp-web-archive@optimus.ietf.org; Mon, 21 Jul 2003 15:35:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08593 for ; Mon, 21 Jul 2003 15:35:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19egR8-0005lJ-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 15:35:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19egR3-0005lG-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 15:35:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19egR3-0002Pa-H1; Mon, 21 Jul 2003 15:35:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19egQX-0002Lm-Au for rddp@optimus.ietf.org; Mon, 21 Jul 2003 15:34:29 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08589 for ; Mon, 21 Jul 2003 15:34:25 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19egQV-0005l1-00 for rddp@ietf.org; Mon, 21 Jul 2003 15:34:27 -0400 Received: from brmea-mail-3.sun.com ([192.18.98.34]) by ietf-mx with esmtp (Exim 4.12) id 19egQL-0005kq-00 for rddp@ietf.org; Mon, 21 Jul 2003 15:34:17 -0400 Received: from phys-aus08-1 ([129.153.131.88]) by brmea-mail-3.sun.com (8.12.9/8.12.9) with ESMTP id h6LJXjCY020389 for ; Mon, 21 Jul 2003 13:33:45 -0600 (MDT) Received: from Sun.COM (jetsun.Central.Sun.COM [129.153.128.60]) by aus08-mail1.central.sun.com (iPlanet Messaging Server 5.2 HotFix 1.10 (built Jan 23 2003)) with ESMTP id <0HIE009BN3O8R5@aus08-mail1.central.sun.com> for rddp@ietf.org; Mon, 21 Jul 2003 14:33:44 -0500 (CDT) Date: Mon, 21 Jul 2003 14:33:44 -0500 From: David Robinson Subject: Re: [rddp] WG chair on buffer exhaustion In-reply-to: <277DD60FB639D511AC0400B0D068B71E0C246750@corpmx14.us.dg.com> To: Black_David@emc.com Cc: rddp@ietf.org Reply-to: David.Robinson@Sun.COM Message-id: <3F1C4018.1000802@Sun.COM> Organization: Sun Microsystems, Inc. MIME-version: 1.0 Content-type: text/plain; format=flowed; charset=us-ascii Content-transfer-encoding: 7bit X-Accept-Language: en-us, en User-Agent: Mozilla/5.0 (X11; U; SunOS sun4u; en-US; rv:1.3) Gecko/20030314 References: <277DD60FB639D511AC0400B0D068B71E0C246750@corpmx14.us.dg.com> Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit I think that David has made a useful observation here. Having the protocol specify that the DDP stream be terminated when an untagged message is presented to DDP when there is a lack of an available buffer is a reasonable specification. What it doesn't prohibit is an implementation from performing tricks to delay delivery which would result in the stream termination. Tricks to delay delivery can include having the transport have a set of hidden buffers, have the transport intentionally fail to ACK or close its window, or even have the bottom layer of DDP have hidden buffering that delays delivery. In a nicely layered approach the semantics are well defined, if you deliver an untagged message when there are not buffers available, the DDP stream MUST be terminated. But in a complex or sophisticated implementation the lower layers are free to perform whatever tricks it wants to delay delivery until buffers are available to prevent termination, but that is a quality of implementation issue. I would venture that it is not worth a discussion of the advantages and disadvantages of such trickery as failing to ACK. Simply state that lower layers MAY delay delivery of an untagged message to DDP when no buffers are available, details are left to the implementation. -David Black_David@emc.com wrote: > The heated debate on whether lack of availability of a buffer for > an untagged message causes immediate termination of the DDP stream > on which it arrived caused some additional email behind the scenes > in addition to what's been on the list. > > With my WG chair hat on, here is what I believe to be the correct > resolution of this issue: > > The request to do something other than immediately terminate > a stream when a message shows up that does not have a pre-queued > untagged buffer to receive it does not require that DDP tell > TCP not to ACK the message; an implementation could choose to > do that based on inter-layer implementation optimizations, > but had better do so very rarely. Alternatively, a DDP > implementation could choose to have some "hidden" resources > in its back pocket that might involve a less efficient receive > path (e.g., receive the errant message to a hidden kernel buffer > and then copy to an application buffer when it becomes available) > - again, this would need to be a rare occurrence for obvious reasons. > This sort of approach is more robust than having TCP not ACK, as > it avoids propagating a host problem into the network and puts the > performance penalty for the implementation problem (should > have had a buffer available, but didn't) squarely on the system > that had the problem. > > I would prefer to see the "hidden [kernel] buffer" approach > described as an example rather than the "TCP doesn't ACK" approach. > Among the other problems with "TCP doesn't ACK" is that it creates > a nasty head-of-line blocking problem that affects all traffic > on the connection, not just the untagged traffic that is facing > the resource shortage. > > OTOH, if the ULP has set a hard limit on the number of untagged > messages that can come in from a specific stream, and that stream > exceeds the limit, this is evidence that the counterparty isn't > following the rules, and termination of the stream is a reasonable > response. In the particular scenario that John raises below, the hard > limit would need to be set high enough to accommodate error/special/ > emergency conditions, and statistical/shared buffer provisioning would > probably be used to avoid having to provision buffers for simultaneous > occurrence of error/special/emergency conditions on all inbound streams. > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > black_david@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > > >>>On Wednesday, July 16, 2003, hufferd@us.ibm.com wrote: >>> >>> >>>>Again, I think that Mallikarjun stated things correctly. There are >>>>important applications which can only correctly predict the number of >>>>buffers needed in non error/special/emergency conditions. When >>>>error/special/emergency conditions occur, the number and urgency of the >>>>handling is problematical. > > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 21 15:48:34 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08884 for ; Mon, 21 Jul 2003 15:48:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19egdj-000308-Jy for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 15:48:07 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6LJm7uZ011537 for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 15:48:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19egdj-000300-3U for rddp-web-archive@optimus.ietf.org; Mon, 21 Jul 2003 15:48:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08847 for ; Mon, 21 Jul 2003 15:48:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19egdh-0005sU-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 15:48:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19egdc-0005sR-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 15:48:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19egdc-0002yI-Iw; Mon, 21 Jul 2003 15:48:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19egd0-0002xQ-N3 for rddp@optimus.ietf.org; Mon, 21 Jul 2003 15:47:22 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA08801 for ; Mon, 21 Jul 2003 15:47:17 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19egcx-0005rY-00 for rddp@ietf.org; Mon, 21 Jul 2003 15:47:19 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19egcn-0005r4-00 for rddp@ietf.org; Mon, 21 Jul 2003 15:47:09 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 71BCB2459CA; Mon, 21 Jul 2003 15:46:22 -0400 (EDT) Date: Mon, 21 Jul 2003 14:46:13 -0500 Subject: Re: [rddp] WG chair on buffer exhaustion Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: Black_David@emc.com, rddp@ietf.org To: David.Robinson@Sun.COM From: Caitlin Bestler In-Reply-To: <3F1C4018.1000802@Sun.COM> Message-Id: Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Monday, July 21, 2003, at 02:33 PM, David Robinson wrote: > I would venture that it is not worth a discussion of the > advantages and disadvantages of such trickery as failing > to ACK. Simply state that lower layers MAY delay delivery > of an untagged message to DDP when no buffers are available, > details are left to the implementation. To the degree that it is an implementation tradeoff, that is true. But I believe it is relevant to simply state along the lines that David has suggested that false drops have an impact on the entire network, and that this solution should be kept to an absolute minimum. It is also important that we recognize that the ULP MUST have some form of flow control, and that *if* that information has been shared with the DDP Layer via the local interface that the DDP Layer not only MAY, but MUST terminate the stream when it receives an untagged segment with an out-of-range MSN. Pat Thaler's comments last week provide some solid reasons as to why it would be a bad idea to *mandate* that this information MUST be shared. I still think it's worthy of a Must Implement, but there were valid application models raised where the local ULP would not want to be bothered with communicating the limit to the DDP layer. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 21 17:26:36 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA11361 for ; Mon, 21 Jul 2003 17:26:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19eiAb-0007dh-Gh for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 17:26:09 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6LLQ9S9029360 for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 17:26:09 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19eiAa-0007ch-Lp for rddp-web-archive@optimus.ietf.org; Mon, 21 Jul 2003 17:26:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA11348 for ; Mon, 21 Jul 2003 17:26:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19eiAY-0006UK-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 17:26:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19eiAS-0006UH-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 17:26:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19eiAU-0007cB-1q; Mon, 21 Jul 2003 17:26:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19eiA4-0007Zq-J8 for rddp@optimus.ietf.org; Mon, 21 Jul 2003 17:25:36 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA11339 for ; Mon, 21 Jul 2003 17:25:31 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19eiA2-0006U6-00 for rddp@ietf.org; Mon, 21 Jul 2003 17:25:34 -0400 Received: from mail4.microsoft.com ([131.107.3.122]) by ietf-mx with esmtp (Exim 4.12) id 19ei9r-0006TS-00 for rddp@ietf.org; Mon, 21 Jul 2003 17:25:23 -0400 Received: from mail5.microsoft.com ([157.54.6.156]) by mail4.microsoft.com with Microsoft SMTPSVC(5.0.2195.6713); Mon, 21 Jul 2003 14:24:01 -0700 Received: from inet-vrs-05.redmond.corp.microsoft.com ([157.54.6.157]) by mail5.microsoft.com with Microsoft SMTPSVC(5.0.2195.6713); Mon, 21 Jul 2003 14:24:00 -0700 Received: from 157.54.5.25 by inet-vrs-05.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Mon, 21 Jul 2003 14:24:00 -0700 Received: from RED-IMC-04.redmond.corp.microsoft.com ([157.54.2.168]) by inet-hub-03.redmond.corp.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Mon, 21 Jul 2003 14:23:22 -0700 Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by RED-IMC-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Mon, 21 Jul 2003 14:23:59 -0700 Received: from WIN-MSG-10.wingroup.windeploy.ntdev.microsoft.com ([157.54.12.81]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Mon, 21 Jul 2003 14:23:38 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------InterScan_NT_MIME_Boundary" Subject: RE: [rddp] security doc - Definition of Trust Date: Mon, 21 Jul 2003 14:23:58 -0700 Message-ID: Thread-Topic: [rddp] security doc - Definition of Trust Thread-Index: AcNPs40RBesy4nRRTPq+y93+sAy6DAAGIQFg From: "Jim Pinkerton" To: "Caitlin Bester" Cc: X-OriginalArrivalTime: 21 Jul 2003 21:23:38.0418 (UTC) FILETIME=[59C0F120:01C34FCE] Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multi-part message in MIME format. --------------InterScan_NT_MIME_Boundary Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C34FCE.5F4EB11F" ------_=_NextPart_001_01C34FCE.5F4EB11F Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I see at least two issues brought up in this email: =20 1) I'm having problems grappling with the various levels of grayness on what the intent of the remote peer is. I think there are many reasons why a particular attack can occur, including: =20 - bug in the implementation - lax coding in an application which runs in kernel space=20 which opens up security holes for other applications - unintentional attacks due to performance optimizations - unintentional attacks due to whatever... - intentional attacks =20 =20 Personally, I don't see it worthwhile to put a lot of text throughout the document saying all the reasons why a particular attack can occur - the most important thing is that it did occur. How it occurred is presumably highly implementation dependent, and thus not necessarily appropriate for a security draft of a wire protocol. =20 In re-reading the security draft though, I realized I don't have this type of statement in the introduction where Trust and Untrusted is defined. Would it help if this was added? =20 Caitlin, could you make a suggestion on recommended text that I could add to the paragraph below on "Untrusted" to point this out? =20 2) Adding of MUST and SHOULD text to the document. To date I've tried to stay away from MUST and SHOULD normative text, and instead point out issues, with recommended solutions. There is exactly one MUST statement in the document: =20 * All Non-Privileged Application interactions with the RNIC Engine that could affect other applications MUST be done using the Privileged Resource Manager as a proxy. The rest of the document takes the approach that if you want this type of a Trust Model, and care about this specific attack, it is RECOMMENDED that you do this. I'm not clear on how to map your issues at the end of the email below to this approach, or are you advocating a different approach in the document? If so, could you outline it? =20 =20 =20 Jim =20 =20 =20 =20 =20 -----Original Message----- From: Caitlin Bester [mailto:caitlinbestler@mac.com]=20 Sent: Monday, July 21, 2003 11:11 AM To: Jim Pinkerton Cc: rddp@ietf.org Subject: Re: [rddp] security doc - Definition of Trust =20 =20 On Monday, July 21, 2003, at 12:44 PM, Jim Pinkerton wrote: =20 > =20 >=20 > One of the issues that has come up on the reflector is that the=20 > definition of Trust should be changed to "Partial Trust", with an=20 > appropriate change to the definition. The existing draft defines trust > as: >=20 > =20 >=20 > Trust - When one party depends upon the other party to not subvert the > goals of the protocols, e.g., it will not attempt to perform the=20 > following attacks: spoofing, repudiation, information disclosure,=20 > denial of service, or elevation of privilege. >=20 > An Untrusted peer is a party that may (or may not) attempt to perform=20 > one or more of the above attacks. A partially trusted peer (either the > Local Peer or Remote Peer) may be trusted to not attempt to perform=20 > some subset of the above attacks, but not trusted to perform a=20 > different subset. >=20 > Possible new definition (feedback/wordsmithing appreciated - I want=20 > general agreement on this before I go through the spec): >=20 > =20 >=20 > Partial Trust - When one party depends upon the other party to not=20 > attempt to perform a specific attack or set of attacks. A partially=20 > trusted peer (either the Local Peer or Remote Peer) may be trusted to=20 > not attempt to perform any subset of the specific list of attacks.=20 > However, there is no guarantee that the Peer won't try to perform a=20 > different set of attacks. >=20 > An Untrusted peer is a party that may (or may not) attempt to perform=20 > one or more of the listed set of attacks. >=20 > Jim >=20 =20 =20 I believe there needs to be a distinction between "trust not to launch=20 an explicit attack" and "trust to conform to the ULP". =20 To a large degree, the intent of the peer is irrelevant. The question=20 is really relates to how paranoid must an application be under what circumstances. =20 For example, while it will always be possible for the ULP to tailor=20 tagged buffer exposure to *exactly* the degree required, a local ULP could easily find this inconvenient to do. If may elect to=20 enable an STag covering a set of buffers, and rely upon its peer to only write to the portion that was enabled per the ULP. =20 However, it should not do so if it imperils any other process or any=20 other party by doing so. =20 If you use a shared buffer pool for untagged messages, and the scope of=20 that sharing encompasses DDP Streams with multiple clients, then your trust of Client A should not place your=20 interactions with Client B at risk (I would almost be tempted to say MUST NOT). =20 Here is a very specific example where Client A may not be "attacking".=20 Client A may merely be a bit over-eager and sending the nTh request before receving the n-mth response and thereby=20 has more than m requests pending. These "cheating clients" have no intention of bringing the server down,=20 but if there are enough of them they will drain the shared buffer pool and eventually force another client's=20 stream to be terminated (or possibly only undergo a false drop). At that point they have damaged the other=20 client, but there was never an intention to "attack". =20 Now just to be safe, it is probably worth stating that there are a set=20 of known "attacks" which are automatically considered to be "non-compliant" with the parent ULP. At least by=20 default, the ULP could be something like a hacker contest where you are *encouraged* to forge your address. But=20 otherwise I think we are quite safe stating that the type of attacks explicitly listed so far are=20 presumed to be non-compliant with the ULP. =20 But the ultimate goal should be to make it clear that: =20 -- The local interface MUST provide tools to allow the ULP to protect=20 itself. -- The local ULP MUST use these tools to enable its peer to perform as=20 specified under the Upper Layer Protocol, -- The local ULP MAY enable more access than what is required to=20 simplify its use of the local interface, but SHOULD NOT do so if it imperils any other process or any other=20 sessions. =20 But the need to protect yourself from stray bullets exists whether the=20 shooter was aiming for you or not. =20 ------_=_NextPart_001_01C34FCE.5F4EB11F Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

I see at least two issues brought up in this = email:

 

1) I'm having problems grappling with the various levels of = grayness on what the intent of the remote peer is. I think there are many reasons = why a particular attack can occur, including:

 

      - bug in the = implementation

      - lax coding = in an application which runs in kernel space

       &nbs= p;    which opens up security holes for other applications

      - = unintentional attacks due to performance optimizations

      - = unintentional attacks due to whatever...

      - intentional = attacks

 

 

Personally, I don't see it worthwhile to put = a lot of text throughout the document saying all the reasons why a particular = attack can occur - the most important thing is that it did occur. How it occurred = is presumably highly implementation dependent, and thus not necessarily appropriate for a security draft of a wire protocol.

 

In re-reading the security draft though, I = realized I don't have this type of statement in the introduction where Trust and = Untrusted is defined. Would it help if this was added?

 

Caitlin, could you make a suggestion on = recommended text that I could add to the paragraph below on "Untrusted" to = point this out?

 

2) Adding of MUST and SHOULD text to the document. To date = I’ve tried to stay away from MUST and SHOULD normative text, and instead = point out issues, with recommended solutions. There is exactly one MUST statement = in the document:

 

*          All Non-Privileged Application interactions = with the RNIC Engine that could affect other applications MUST be done using = the Privileged Resource Manager as a proxy.

The rest of the document takes the approach = that if you want this type of a Trust Model, and care about this specific = attack, it is RECOMMENDED that you do this. I’m not clear on how to map your = issues at the end of the email below to this approach, or are you advocating a = different approach in the document? If so, could you outline it?

 

 

 

Jim

 

 

 

 

 

-----Original Message-----
From: Caitlin Bester [mailto:caitlinbestler@mac.com]
Sent: Monday, July 21, 2003 11:11 AM
To: Jim Pinkerton
Cc: rddp@ietf.org
Subject: Re: [rddp] security doc - Definition of Trust

 

 

On Monday, July 21, 2003, at 12:44 PM, Jim Pinkerton = wrote:

 

>  

> 

> One of the issues that has come up on the reflector is that = the

> definition of Trust should be changed to “Partial Trust”, with an

> appropriate change to the definition. The existing draft = defines trust

> as:

> 

>  

> 

> Trust - When one party depends upon the other party to not = subvert the

> goals of the protocols, e.g., it will not attempt to = perform the

> following attacks: spoofing, repudiation, information = disclosure,

> denial of service, or elevation of = privilege.

> 

> An Untrusted peer is a party that may (or may not) attempt = to perform

> one or more of the above attacks. A partially trusted peer = (either the

> Local Peer or Remote Peer) may be trusted to not attempt to perform

> some subset of the above attacks, but not trusted to = perform a

> different subset.

> 

> Possible new definition (feedback/wordsmithing appreciated = – I want

> general agreement on this before I go through the = spec):

> 

>  

> 

> Partial Trust - When one party depends upon the other party = to not

> attempt to perform a specific attack or set of attacks. A partially

> trusted peer (either the Local Peer or Remote Peer) may be = trusted to

> not attempt to perform any subset of the specific list of = attacks.

> However, there is no guarantee that the Peer won’t = try to perform a

> different set of attacks.

> 

> An Untrusted peer is a party that may (or may not) attempt = to perform

> one or more of the listed set of attacks.

> 

> Jim

> 

 

 

I believe there needs to be a distinction between "trust = not to launch

an explicit attack" and "trust to conform to the = ULP".

 

To a large degree, the intent of the peer is irrelevant. The = question

is really relates to how paranoid must an = application

be under what circumstances.

 

For example, while it will always be possible for the ULP to = tailor

tagged buffer exposure to *exactly* the degree = required,

a local ULP could easily find this inconvenient to do. If may = elect to

enable an STag covering a set of buffers, and = rely

upon its peer to only write to the portion that was enabled per = the ULP.

 

However, it should not do so if it imperils any other process or = any

other party by doing so.

 

If you use a shared buffer pool for untagged messages, and the = scope of

that sharing encompasses DDP Streams with

multiple clients, then your trust of Client A should not place = your

interactions with Client B at risk (I would almost = be

tempted to say MUST NOT).

 

Here is a very specific example where Client A may not be "attacking".

Client A may merely be a bit over-eager and

sending the nTh request before receving the n-mth response and = thereby

has more than m requests pending.

These "cheating clients" have no intention of bringing = the server down,

but if there are enough of them they will

drain the shared buffer pool and eventually force another = client's

stream to be terminated (or possibly only

undergo a false drop). At that point they have damaged the other =

client, but there was never an intention to = "attack".

 

Now just to be safe, it is probably worth stating that there are = a set

of known "attacks" which are = automatically

considered to be "non-compliant" with the parent ULP. = At least by

default, the ULP could be something like

a hacker contest where you are *encouraged* to forge your = address. But

otherwise I think we are quite

safe stating that the type of attacks explicitly listed so far = are

presumed to be non-compliant with the ULP.

 

But the ultimate goal should be to make it clear = that:

 

-- The local interface MUST provide tools to allow the ULP to = protect

itself.

-- The local ULP  MUST use these tools to enable its peer = to perform as

specified under the Upper Layer Protocol,

-- The local ULP MAY enable more access than what is required to =

simplify its use of the local interface, but

     SHOULD NOT do so if it imperils any = other process or any other

sessions.

 

But the need to protect yourself from stray bullets exists = whether the

shooter was aiming for you or not.

 

------_=_NextPart_001_01C34FCE.5F4EB11F-- --------------InterScan_NT_MIME_Boundary-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 21 18:23:33 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA13576 for ; Mon, 21 Jul 2003 18:23:33 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ej3k-00014a-Uq for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 18:23:09 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6LMN8lR004120 for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 18:23:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ej3k-00014N-Ke for rddp-web-archive@optimus.ietf.org; Mon, 21 Jul 2003 18:23:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA13573 for ; Mon, 21 Jul 2003 18:23:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ej3h-0006kM-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 18:23:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ej3c-0006kJ-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 18:23:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ej3d-00013t-2Q; Mon, 21 Jul 2003 18:23:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ej3K-00013h-6X for rddp@optimus.ietf.org; Mon, 21 Jul 2003 18:22:42 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA13569 for ; Mon, 21 Jul 2003 18:22:35 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ej3G-0006kF-00 for rddp@ietf.org; Mon, 21 Jul 2003 18:22:38 -0400 Received: from emulex.emulex.com ([138.239.112.1]) by ietf-mx with esmtp (Exim 4.12) id 19ej36-0006k9-00 for rddp@ietf.org; Mon, 21 Jul 2003 18:22:28 -0400 Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [138.239.73.11]) by emulex.emulex.com (8.12.9/8.12.8) with ESMTP id h6LMLbpJ018166 for ; Mon, 21 Jul 2003 15:21:37 -0700 (PDT) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id <3V4RZVRQ>; Mon, 21 Jul 2003 18:21:36 -0400 Message-ID: <3356669BBE90C448AD4645C843E2BF28020C8266@xbl.ma.emulex.com> From: "Williams, Jim" To: rddp@ietf.org Subject: RE: [rddp] CRC rough consensus from Vienna Date: Mon, 21 Jul 2003 18:19:57 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Unfortunately, I was not able to make it to Vienna, so my comments below may echo the discussion that already took place. > From: Black_David@emc.com [mailto:Black_David@emc.com] > > The discussion that is the basis for this rough consensus > acknowledges that there are some advantages to the split CRCs, > principally that the header CRC can be verified without waiting > for the data, especially if the data spans more than one packet. > The principal downside is increased complexity of two CRCs. > The advantages of two CRCs are not overwhelming, as for aligned > traffic, the data and its CRC will arrive immediately, and in > the unaligned case, an implementation that wants to use the > header for immediate placement without waiting for a second > packet will be able to verify the TCP checksum before placement. If it is acceptable for an implementation to place data based on a header that has been validated only with TCP checksum (not yet with CRC), then I am fine with this approach. Emphasis on "acceptable", not just "allowable". > It is also ok in an error case to tell the ULP that "something > went wrong, the data in the buffer you handed to RDDP for placement > is invalid as it may have been overwritten in part" - the TCP > connection on which this happened will generally be terminated > in this case. OK, but this still may be a serious error. Theoretically, a buffer may be completed, and returned to the ULP. While the ULP is using it, it is erroneously overwritten. The ULP uses the corrupt data. Only later when the CRC arrives is the ULP notified that the data was bad. On the other hand, if there is a TCP checksum escape, the conditional probability that an error occurred in the RDDP header STag or offset, AND that the header is still valid and passes protection checks, AND that the buffer is currently being used by the ULP is low. So the protection is much better than the TCP checksum alone. For writes that appear untagged, the ULP will not see the data until the CRC has been checked, so no problem. > This does not appear to rule out or unfairly disadvantage any > important classes of implementations. My recollection of past > discussion is that I need to call this rough consensus over Jim > Williams's objections, but comments from Jim and anyone else who > takes issue with this position should be sent to the list. I believe the above has satisfactorily addressed any issues that I have. > Thanks, > --David _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 21 18:48:32 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA14076 for ; Mon, 21 Jul 2003 18:48:32 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ejRv-0001uP-La for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 18:48:08 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6LMm7gm007331 for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 18:48:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ejRv-0001uA-H3 for rddp-web-archive@optimus.ietf.org; Mon, 21 Jul 2003 18:48:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA14059 for ; Mon, 21 Jul 2003 18:48:00 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ejRs-0006r7-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 18:48:04 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ejRm-0006r4-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 18:47:58 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ejRo-0001te-G7; Mon, 21 Jul 2003 18:48:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ejR1-0001re-2n for rddp@optimus.ietf.org; Mon, 21 Jul 2003 18:47:11 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA14054 for ; Mon, 21 Jul 2003 18:47:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ejQx-0006r1-00 for rddp@ietf.org; Mon, 21 Jul 2003 18:47:07 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19ejQn-0006qu-00 for rddp@ietf.org; Mon, 21 Jul 2003 18:46:57 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 7F6BF244F72; Mon, 21 Jul 2003 18:46:13 -0400 (EDT) Date: Mon, 21 Jul 2003 17:46:04 -0500 Subject: Re: [rddp] security doc - Definition of Trust Content-Type: text/plain; charset=WINDOWS-1252; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: To: "Jim Pinkerton" From: Caitlin Bestler In-Reply-To: Message-Id: <1C5DCCF6-BBCD-11D7-A23F-003065D48EE0@asomi.com> Content-Transfer-Encoding: quoted-printable X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable On Monday, July 21, 2003, at 04:23 PM, Jim Pinkerton wrote: > 1) I'm having problems grappling with the various levels of grayness=20= > on what the intent of the remote peer is. I think there are many=20 > reasons why a particular attack can occur, including: > > =A0 > > =A0=A0=A0=A0=A0 - bug in the implementation > > =A0=A0=A0=A0=A0 - lax coding in an application which runs in kernel = space > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 which opens up security holes for = other applications > > =A0=A0=A0=A0=A0 - unintentional attacks due to performance = optimizations > > =A0=A0=A0=A0=A0 - unintentional attacks due to whatever... > > =A0=A0=A0=A0=A0 - intentional attacks > > =A0 > > =A0 > > Personally, I don't see it worthwhile to put a lot of text throughout=20= > the document saying all the reasons why a particular attack can occur=20= > - the most important thing is that it did occur. How it occurred is=20 > presumably highly implementation dependent, and thus not necessarily=20= > appropriate for a security draft of a wire protocol. > > =A0 I prefer to simply describe such behavior by the peer as=20 "non-compliant". It places the emphasis on the fact that the buffer access varies from what the Upper Layer Protocol=20 specified, without dwelling on why or how it is varying. It also places the burden on the local ULP. If you choose to enable one=20= giant STag, and then simply advertise three buffers within it, you need to be aware that you are=20 trusting your peer to be ULP-compliant. There can be major advantages to granting such trust, such as=20 simplicity of coding. But you need to determine what you have placed at risk. I believe we can safely state that there are two cases where the local=20= ULP SHOULD NOT do this (or at least that it is really heavily recommended that they do not): -- Exposure to resources that do not belong to the local ULP. -- Exposure of resources that other clients / sessions rely upon. The latter is more subtle, it is not necessarily resources that can be=20= accessed on other DDP Stream. The local client could have multiple connections to the *same* remote=20 partner (especially if they still haven't discovered SCTP and insist on duplicating its features using=20 TCP). Or the remote partner may be expressed as members of a cluster. In either case the DDP Streams=20 that are being placed "at risk" are all supporting the same remote entity. An application should feel=20 free to determine how much double-checking it wants to do against remote entity violations as part=20= of being robust. Being robust in the face peer errors is certainly commendable, but it is not=20 something that should be mandated or even lobbied for. That is strictly a decision for each application to=20 determine how robust it will be. But once you allow resource sharing across DDP Streams that support=20 *different* remote entities, you are leaving your peers subject to an indirect Denial of Service attack.=20= That is a totally different issue. > > In re-reading the security draft though, I realized I don't have this=20= > type of statement in the introduction where Trust and Untrusted is=20 > defined. Would it help if this was added? > > =A0 > > Caitlin, could you make a suggestion on recommended text that I could=20= > add to the paragraph below on "Untrusted" to point this out? > An "untrusted" peer is any that you do not want to rely upon to be ULP=20= compliant. A "trusted" peer is one that you do rely upon to be ULP compliant, even=20= if the only reason for doing so is that you believe your handling of that peer is sufficiently "walled=20 off" that the scope of potential damage is deemed acceptable. That is, the peer can only shoot themselves in=20 the foot. > > 2) Adding of MUST and SHOULD text to the document. To date I=92ve = tried=20 > to stay away from MUST and SHOULD normative text, and instead point=20 > out issues, with recommended solutions. There is exactly one MUST=20 > statement in the document: > > =A0 > > *=A0=A0=A0=A0=A0=A0=A0=A0=A0All Non-Privileged Application = interactions with the RNIC=20 > Engine that could affect other applications MUST be done using the=20 > Privileged Resource Manager as a proxy. > I do have a recommendation here. A "Privileged Resource Manager" is a=20 specific strategy. A common one, true, but it is not universal. A more generic statement would be: The local interface MUST NOT allow an application to grant the RNIC=20= Engine any rights or access privileges that it does not itself already have. Typically this will require=20 certain requests to be proxied by a Privileged Resource Manager or other mechanism that allows the RNIC=20= Engine to verify access rights. > The rest of the document takes the approach that if you want this type=20= > of a Trust Model, and care about this specific attack, it is=20 > RECOMMENDED that you do this. I=92m not clear on how to map your = issues=20 > at the end of the email below to this approach, or are you advocating=20= > a different approach in the document? If so, could you outline it? > Ok, let's give it a try. My outline would run something like this. 1 - Enabling the RNIC Engine to place memory directly to ULP buffers=20 requires privilege The local interface must have some mechanism by which the RNIC = Engine=20 verifies that its clients have authority to make the requests being made. This may rely upon a=20 single Privileged Resource Manager to proxy critical authorization requests. 2 - If your peer is compliant with the Upper Layer Protocol then any=20 security issues are dealt with elsewhere. In other words, if you're accepting text messages which you pass = to a=20 privileged shell for execution, your security problems are not an RDMAP/DDP issue. Except as noted for rule #1, local OS security rules are also = not an=20 RDMAP/DDP issue. If an OS does not have adequate security for non-RDMA applications then RDMAP/DDP=20 will not solve any of those problems. 3 - You authorize your peer to make certain remote accesses to untagged=20= and/or tagged buffers. How you communicate this is out-of-scope. However you must = enable the=20 RNIC Engine to accept all remote access that you authorize. At this point the emphasis should be on the fact that if the = local ULP=20 uses the local interface with precision that the remote peer will be incapable of doing any=20= damage. An RDMA application can actually be less vulnerable to attack than one using the LLP=20 directly. 4 - You may end up enabling more access than is ideal -- Potential=20 impacts and counter-measures Examples of this include the time gap from the last tagged put=20 until you can invalidate the STag, or using a overly broad STag that enables multiple advertised=20 buffers. The potential impacts should be explained and highlighted. (The=20 local ULP cannot detect excess or inappropriate Tagged puts unless remote access is invalidated.=20= Unlike a classic buffer overrun attack, there would not normally be logged information allow an ex=20= post facto diagnosis). Methods of avoiding the attack would be highlighted (use narrow=20 STags that are quickly invalidated, check the buffer only after invalidating remote access, etc.) The current draft deals with these very well for tagged buffers. = It is=20 the discussions on untagged messages and shared resources that I currently have problems with. It is particularly in the context of impact on shared resources = that=20 the distinctions on "ULP compliance" and "who is impacted" become critical. If my local=20= interface allows me to share buffers and completion queues, but still allows me to impose a strict limit=20= on the number of untagged messages that will be accepted from my peer, then there is no=20 exposure. The peer is ULP-compliant, or the DDP Stream is terminated. If I have a soft limit on untagged messages once shared resources=20= are in use, then I have a potential vulnerability. I need a strategy to determine how much=20 could be exposed to a non-compliant remote peer in order to ensure that there were adequate safeguards=20= to prevent one faulty/attacking peer from adversely impacting others. Too much of the current outline seems to imply that you can be=20 paranoid or you can be efficient, without really helping the application developer find their own=20 optimum balance. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 21 21:02:35 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA16423 for ; Mon, 21 Jul 2003 21:02:35 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19elXd-000666-3g for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 21:02:09 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6M129Zq023432 for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 21:02:09 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19elXc-00065r-91 for rddp-web-archive@optimus.ietf.org; Mon, 21 Jul 2003 21:02:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA16408 for ; Mon, 21 Jul 2003 21:02:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19elXZ-0007jU-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 21:02:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19elXU-0007jR-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 21:02:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19elXV-00064A-HT; Mon, 21 Jul 2003 21:02:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19elWY-00063s-K2 for rddp@optimus.ietf.org; Mon, 21 Jul 2003 21:01:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA16366 for ; Mon, 21 Jul 2003 21:00:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19elWW-0007is-00 for rddp@ietf.org; Mon, 21 Jul 2003 21:01:00 -0400 Received: from mail1.microsoft.com ([131.107.3.125]) by ietf-mx with esmtp (Exim 4.12) id 19elWL-0007ht-00 for rddp@ietf.org; Mon, 21 Jul 2003 21:00:49 -0400 Received: from inet-vrs-01.redmond.corp.microsoft.com ([157.54.8.27]) by mail1.microsoft.com with Microsoft SMTPSVC(5.0.2195.6713); Mon, 21 Jul 2003 17:58:59 -0700 Received: from 157.54.6.150 by inet-vrs-01.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Mon, 21 Jul 2003 17:58:58 -0700 Received: from RED-IMC-04.redmond.corp.microsoft.com ([157.54.2.168]) by inet-hub-05.redmond.corp.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Mon, 21 Jul 2003 17:59:09 -0700 Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by RED-IMC-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Mon, 21 Jul 2003 17:58:54 -0700 Received: from WIN-MSG-10.wingroup.windeploy.ntdev.microsoft.com ([157.54.12.81]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Mon, 21 Jul 2003 17:58:57 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Subject: RE: [rddp] security doc - Definition of Trust Date: Mon, 21 Jul 2003 17:58:57 -0700 Message-ID: Thread-Topic: [rddp] security doc - Definition of Trust Thread-Index: AcNP2ee6lIVThnYnR6KvxyRxZfOkEwAEH3QA From: "Jim Pinkerton" To: "Caitlin Bestler" Cc: X-OriginalArrivalTime: 22 Jul 2003 00:58:57.0486 (UTC) FILETIME=[6E1E46E0:01C34FEC] Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable I understand the point you're trying to make about being "ULP = Compliant", but personally find the term too in-exact. To me we aren't = trying to define a ULP here - we're simply identifying the things that = any ULP can use in the DDP/RDMAP protocol, and security issues/attacks = that those features can expose if used.=20 What the authors were attempting to do was create a short hand for = application developers to use - is your application model that you can = make no assumptions about the remote peer, and thus have to assume the = worst (i.e. typical networking), or is it the logical exact opposite, = where everyone trusts everyone (which is typical for clustering - a = single fault in an element of the cluster causes the entire clustered = application to die). And then the shades of grey in the middle (local = vs. remote trust, etc).=20 I agree with your assessment that the current draft nails the tagged = buffer model pretty well, but the untagged model - specifically the = shared pool of untagged buffers (whichever algorithm you choose to = implement this) is not dealt with at all. This is an action item taken = away from Vienna. Another point you make below is that multiple clients to a single server = analysis should be better threshed out. Will do. The draft does not deal = with it explicitly today, and should. On the idea of removing the privileged resource manager, I guess I'm = unclear on the advantage of it. It's just a reference model to help = clarify issues. There are no MUSTs anywhere. Using the text as you = describe basically collapses the functionality into the RNIC. To me one = of the intentions of the paper was to specifically slice up the security = issues into more detail to better explain the issues. How you implement = it is your business though. On the last 4 points you make, I'll need to walk through the draft more = thoroughly to see how they map on. Looks like some good ideas. Jim -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com]=20 Sent: Monday, July 21, 2003 3:46 PM To: Jim Pinkerton Cc: rddp@ietf.org Subject: Re: [rddp] security doc - Definition of Trust On Monday, July 21, 2003, at 04:23 PM, Jim Pinkerton wrote: > 1) I'm having problems grappling with the various levels of grayness=20 > on what the intent of the remote peer is. I think there are many=20 > reasons why a particular attack can occur, including: > > =A0 > > =A0=A0=A0=A0=A0 - bug in the implementation > > =A0=A0=A0=A0=A0 - lax coding in an application which runs in kernel = space > > =A0=A0=A0=A0=A0=A0=A0=A0=A0=A0=A0 which opens up security holes for = other applications > > =A0=A0=A0=A0=A0 - unintentional attacks due to performance = optimizations > > =A0=A0=A0=A0=A0 - unintentional attacks due to whatever... > > =A0=A0=A0=A0=A0 - intentional attacks > > =A0 > > =A0 > > Personally, I don't see it worthwhile to put a lot of text throughout=20 > the document saying all the reasons why a particular attack can occur=20 > - the most important thing is that it did occur. How it occurred is=20 > presumably highly implementation dependent, and thus not necessarily=20 > appropriate for a security draft of a wire protocol. > > =A0 I prefer to simply describe such behavior by the peer as=20 "non-compliant". It places the emphasis on the fact that the buffer access varies from what the Upper Layer Protocol=20 specified, without dwelling on why or how it is varying. It also places the burden on the local ULP. If you choose to enable one=20 giant STag, and then simply advertise three buffers within it, you need to be aware that you are=20 trusting your peer to be ULP-compliant. There can be major advantages to granting such trust, such as=20 simplicity of coding. But you need to determine what you have placed at risk. I believe we can safely state that there are two cases where the local=20 ULP SHOULD NOT do this (or at least that it is really heavily recommended that they do not): -- Exposure to resources that do not belong to the local ULP. -- Exposure of resources that other clients / sessions rely upon. The latter is more subtle, it is not necessarily resources that can be=20 accessed on other DDP Stream. The local client could have multiple connections to the *same* remote=20 partner (especially if they still haven't discovered SCTP and insist on duplicating its features using=20 TCP). Or the remote partner may be expressed as members of a cluster. In either case the DDP Streams=20 that are being placed "at risk" are all supporting the same remote entity. An application should feel=20 free to determine how much double-checking it wants to do against remote entity violations as part=20 of being robust. Being robust in the face peer errors is certainly commendable, but it is not=20 something that should be mandated or even lobbied for. That is strictly a decision for each application to=20 determine how robust it will be. But once you allow resource sharing across DDP Streams that support=20 *different* remote entities, you are leaving your peers subject to an indirect Denial of Service attack.=20 That is a totally different issue. > > In re-reading the security draft though, I realized I don't have this=20 > type of statement in the introduction where Trust and Untrusted is=20 > defined. Would it help if this was added? > > =A0 > > Caitlin, could you make a suggestion on recommended text that I could=20 > add to the paragraph below on "Untrusted" to point this out? > An "untrusted" peer is any that you do not want to rely upon to be ULP=20 compliant. A "trusted" peer is one that you do rely upon to be ULP compliant, even=20 if the only reason for doing so is that you believe your handling of that peer is sufficiently "walled=20 off" that the scope of potential damage is deemed acceptable. That is, the peer can only shoot themselves in=20 the foot. > > 2) Adding of MUST and SHOULD text to the document. To date I've tried=20 > to stay away from MUST and SHOULD normative text, and instead point=20 > out issues, with recommended solutions. There is exactly one MUST=20 > statement in the document: > > =A0 > > *=A0=A0=A0=A0=A0=A0=A0=A0=A0All Non-Privileged Application = interactions with the RNIC=20 > Engine that could affect other applications MUST be done using the=20 > Privileged Resource Manager as a proxy. > I do have a recommendation here. A "Privileged Resource Manager" is a=20 specific strategy. A common one, true, but it is not universal. A more generic statement would be: The local interface MUST NOT allow an application to grant the RNIC=20 Engine any rights or access privileges that it does not itself already have. Typically this will require=20 certain requests to be proxied by a Privileged Resource Manager or other mechanism that allows the RNIC=20 Engine to verify access rights. > The rest of the document takes the approach that if you want this type = > of a Trust Model, and care about this specific attack, it is=20 > RECOMMENDED that you do this. I'm not clear on how to map your issues=20 > at the end of the email below to this approach, or are you advocating=20 > a different approach in the document? If so, could you outline it? > Ok, let's give it a try. My outline would run something like this. 1 - Enabling the RNIC Engine to place memory directly to ULP buffers=20 requires privilege The local interface must have some mechanism by which the RNIC Engine=20 verifies that its clients have authority to make the requests being made. This may rely upon a=20 single Privileged Resource Manager to proxy critical authorization requests. 2 - If your peer is compliant with the Upper Layer Protocol then any=20 security issues are dealt with elsewhere. In other words, if you're accepting text messages which you pass to a=20 privileged shell for execution, your security problems are not an RDMAP/DDP issue. Except as noted for rule #1, local OS security rules are also not an=20 RDMAP/DDP issue. If an OS does not have adequate security for non-RDMA applications then RDMAP/DDP=20 will not solve any of those problems. 3 - You authorize your peer to make certain remote accesses to untagged=20 and/or tagged buffers. How you communicate this is out-of-scope. However you must enable the=20 RNIC Engine to accept all remote access that you authorize. At this point the emphasis should be on the fact that if the local ULP=20 uses the local interface with precision that the remote peer will be incapable of doing any=20 damage. An RDMA application can actually be less vulnerable to attack than one using the LLP=20 directly. 4 - You may end up enabling more access than is ideal -- Potential=20 impacts and counter-measures Examples of this include the time gap from the last tagged put=20 until you can invalidate the STag, or using a overly broad STag that enables multiple advertised=20 buffers. The potential impacts should be explained and highlighted. (The=20 local ULP cannot detect excess or inappropriate Tagged puts unless remote access is invalidated.=20 Unlike a classic buffer overrun attack, there would not normally be logged information allow an ex=20 post facto diagnosis). Methods of avoiding the attack would be highlighted (use narrow=20 STags that are quickly invalidated, check the buffer only after invalidating remote access, etc.) The current draft deals with these very well for tagged buffers. It is=20 the discussions on untagged messages and shared resources that I currently have problems with. It is particularly in the context of impact on shared resources that=20 the distinctions on "ULP compliance" and "who is impacted" become critical. If my local=20 interface allows me to share buffers and completion queues, but still allows me to impose a strict limit = on the number of untagged messages that will be accepted from my peer, then there is no=20 exposure. The peer is ULP-compliant, or the DDP Stream is terminated. If I have a soft limit on untagged messages once shared resources=20 are in use, then I have a potential vulnerability. I need a strategy to determine how much=20 could be exposed to a non-compliant remote peer in order to ensure that there were adequate safeguards=20 to prevent one faulty/attacking peer from adversely impacting others. Too much of the current outline seems to imply that you can be=20 paranoid or you can be efficient, without really helping the application developer find their own=20 optimum balance. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 21 21:15:32 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA16647 for ; Mon, 21 Jul 2003 21:15:32 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19elkB-0006b2-3g for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 21:15:07 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6M1F7v0025350 for rddp-archive@odin.ietf.org; Mon, 21 Jul 2003 21:15:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19elkB-0006an-0V for rddp-web-archive@optimus.ietf.org; Mon, 21 Jul 2003 21:15:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA16634 for ; Mon, 21 Jul 2003 21:15:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19elk8-00001X-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 21:15:04 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19elk2-00001U-00 for rddp-web-archive@ietf.org; Mon, 21 Jul 2003 21:14:58 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19elk4-0006ZD-MS; Mon, 21 Jul 2003 21:15:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19eljO-0006Yd-Sm for rddp@optimus.ietf.org; Mon, 21 Jul 2003 21:14:18 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA16622 for ; Mon, 21 Jul 2003 21:14:14 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19eljM-000013-00 for rddp@ietf.org; Mon, 21 Jul 2003 21:14:16 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19eljB-00000j-00 for rddp@ietf.org; Mon, 21 Jul 2003 21:14:05 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 3D3D824598B; Mon, 21 Jul 2003 21:12:45 -0400 (EDT) Date: Mon, 21 Jul 2003 20:12:35 -0500 Subject: Re: [rddp] security doc - Definition of Trust Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: To: "Jim Pinkerton" From: Caitlin Bestler In-Reply-To: Message-Id: <93E77B0E-BBE1-11D7-AF8D-003065D48EE0@asomi.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Monday, July 21, 2003, at 07:58 PM, Jim Pinkerton wrote: > On the idea of removing the privileged resource manager, I guess I'm > unclear on the advantage of it. It's just a reference model to help > clarify issues. There are no MUSTs anywhere. Using the text as you > describe basically collapses the functionality into the RNIC. To me > one of the intentions of the paper was to specifically slice up the > security issues into more detail to better explain the issues. How you > implement it is your business though. I'm trying to be inclusive of all possible implementations. For example, RDMAP/DDP could be implemented as a user-mode library running on top of a traditional stack. If it only exposed the applications own memory then there is no need for a "Privileged Resource Manager". The looser rule, that the local interface grant the RNIC Engine no authority that the caller itself did not have, is met by simply running within the process. At the opposite extreme, I have worked with system buses where the driver was allowed to directly DMA into a user map. (Not that anyone has done anything like that *recently*, but ideas have a way of cycling back around. Who knows about 3 years from now?). So I'd like to recognize that there are multiple ways of ensuring that an RNIC Engine does not violate the host system's security rules. The descriptions of the Privileged Resource Manager presume a multi-process Unix/Windows style OS dealing with an RNIC Engine implemented primarily in hardware and/or fully enabled (i.e., kernel) memory space. That's certainly 99.99 whatever % of the world, but it isn't everything. Perhaps if the principle were established that some form of access validation would be required, and that it would likely be required to start from a single root authorizer. The Privileged Resource Manager described could then be given as an example. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 22 09:32:35 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA12962 for ; Tue, 22 Jul 2003 09:32:35 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19exFQ-00078v-JC for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 09:32:08 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6MDW867027453 for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 09:32:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19exFQ-00078i-G8 for rddp-web-archive@optimus.ietf.org; Tue, 22 Jul 2003 09:32:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA12938 for ; Tue, 22 Jul 2003 09:32:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19exFO-0004dK-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 09:32:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19exFJ-0004dH-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 09:32:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19exFJ-00078E-LE; Tue, 22 Jul 2003 09:32:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19exEl-00077p-V4 for rddp@optimus.ietf.org; Tue, 22 Jul 2003 09:31:27 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA12933 for ; Tue, 22 Jul 2003 09:31:24 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19exEk-0004d0-00 for rddp@ietf.org; Tue, 22 Jul 2003 09:31:26 -0400 Received: from ztxmail05.ztx.compaq.com ([161.114.1.209]) by ietf-mx with esmtp (Exim 4.12) id 19exEK-0004cX-00 for rddp@ietf.org; Tue, 22 Jul 2003 09:31:00 -0400 Received: from cceexg13.americas.cpqcorp.net (cceexg13.americas.cpqcorp.net [16.110.250.119]) by ztxmail05.ztx.compaq.com (Postfix) with ESMTP id EE594E349; Tue, 22 Jul 2003 08:29:54 -0500 (CDT) Received: from cceexc18.americas.cpqcorp.net ([16.81.1.4]) by cceexg13.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Tue, 22 Jul 2003 08:29:54 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 Content-Class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [rddp] WG chair on buffer exhaustion Date: Tue, 22 Jul 2003 08:29:53 -0500 Message-ID: <4D027986353D1341ADA4F2A4F8A170762AB1E92E@cceexc18.americas.cpqcorp.net> Thread-Topic: [rddp] WG chair on buffer exhaustion Thread-Index: AcNPvI7pWIo/wIu4R3SU753keuDX/AAlFkVw From: "Culley, Paul" To: , X-OriginalArrivalTime: 22 Jul 2003 13:29:54.0619 (UTC) FILETIME=[564328B0:01C35055] Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable Do I read into this that you believe that DDP implementations MUST implement a per stream hard limit on untagged messages ("termination is a reasonable response")? If you do, the next question is how fast? Is it ok to delay the stream termination under some conditions, until some "extra" untagged messages have arrived and used buffers? This might occur if the mechanism for detecting the hard limit exception depended on counting completions in software, for example, or if it depended on software to respond to an exception. As pointed out earlier in this thread, the mechanism that returns "credits" for the "hard limit" can become an additional burden for implementations, depending on how that is done. A "loose" credit check and/or update can significantly reduce that burden. =20 It is not clear how much guidance the DDP security specs need to provide on this issue, other than to say something to the effect of: "Here is an issue, make sure that your implementation, or upper interface and ULP can deal with it." Seems like getting too detailed may provide unnecessary restrictions on implementations. Paul R. Culley HP Senior Fellow 281-514-5543 > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com]=20 > Sent: Monday, July 21, 2003 2:03 PM > To: rddp@ietf.org > Subject: [rddp] WG chair on buffer exhaustion >=20 >=20 > The heated debate on whether lack of availability of a buffer=20 > for an untagged message causes immediate termination of the=20 > DDP stream on which it arrived caused some additional email=20 > behind the scenes in addition to what's been on the list. >=20 > With my WG chair hat on, here is what I believe to be the=20 > correct resolution of this issue: >=20 > The request to do something other than immediately terminate > a stream when a message shows up that does not have a=20 > pre-queued untagged buffer to receive it does not require=20 > that DDP tell TCP not to ACK the message; an implementation=20 > could choose to do that based on inter-layer implementation=20 > optimizations, but had better do so very rarely. =20 > Alternatively, a DDP implementation could choose to have some=20 > "hidden" resources in its back pocket that might involve a=20 > less efficient receive path (e.g., receive the errant message=20 > to a hidden kernel buffer and then copy to an application=20 > buffer when it becomes available) > - again, this would need to be a rare occurrence for obvious=20 > reasons. This sort of approach is more robust than having TCP=20 > not ACK, as it avoids propagating a host problem into the=20 > network and puts the performance penalty for the=20 > implementation problem (should have had a buffer available,=20 > but didn't) squarely on the system that had the problem. >=20 > I would prefer to see the "hidden [kernel] buffer" approach=20 > described as an example rather than the "TCP doesn't ACK"=20 > approach. Among the other problems with "TCP doesn't ACK" is=20 > that it creates a nasty head-of-line blocking problem that=20 > affects all traffic on the connection, not just the untagged=20 > traffic that is facing the resource shortage. >=20 > OTOH, if the ULP has set a hard limit on the number of=20 > untagged messages that can come in from a specific stream,=20 > and that stream exceeds the limit, this is evidence that the=20 > counterparty isn't following the rules, and termination of=20 > the stream is a reasonable response. In the particular=20 > scenario that John raises below, the hard limit would need to=20 > be set high enough to accommodate error/special/ emergency=20 > conditions, and statistical/shared buffer provisioning would=20 > probably be used to avoid having to provision buffers for=20 > simultaneous occurrence of error/special/emergency conditions=20 > on all inbound streams. >=20 > Thanks, > --David > ---------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > black_david@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- >=20 > > > > > >On Wednesday, July 16, 2003, hufferd@us.ibm.com wrote: > > > > > >>Again, I think that Mallikarjun stated things correctly. =20 > There are=20 > > >>important applications which can only correctly predict=20 > the number=20 > > >>of buffers needed in non error/special/emergency=20 > conditions. When=20 > > >>error/special/emergency conditions occur, the number and=20 > urgency of=20 > > >>the handling is problematical. >=20 > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp >=20 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 22 10:01:35 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA13640 for ; Tue, 22 Jul 2003 10:01:35 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19exhV-0008A2-Jf for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 10:01:09 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6ME19ii031369 for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 10:01:09 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19exhV-00089s-Gv for rddp-web-archive@optimus.ietf.org; Tue, 22 Jul 2003 10:01:09 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA13618 for ; Tue, 22 Jul 2003 10:01:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19exhS-0004oj-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 10:01:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19exhM-0004og-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 10:01:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19exhN-00087m-SA; Tue, 22 Jul 2003 10:01:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19exgp-00086y-0a for rddp@optimus.ietf.org; Tue, 22 Jul 2003 10:00:27 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA13601 for ; Tue, 22 Jul 2003 10:00:22 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19exgm-0004oT-00 for rddp@ietf.org; Tue, 22 Jul 2003 10:00:24 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19exgc-0004nO-00 for rddp@ietf.org; Tue, 22 Jul 2003 10:00:14 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 19CD1244AA0; Tue, 22 Jul 2003 09:58:53 -0400 (EDT) Date: Tue, 22 Jul 2003 08:58:42 -0500 Subject: Re: [rddp] WG chair on buffer exhaustion Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: , To: "Culley, Paul" From: Caitlin Bestler In-Reply-To: <4D027986353D1341ADA4F2A4F8A170762AB1E92E@cceexc18.americas.cpqcorp.net> Message-Id: <9A5BDE40-BC4C-11D7-AE83-003065D48EE0@asomi.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Tuesday, July 22, 2003, at 08:29 AM, Culley, Paul wrote: > As pointed out earlier in this thread, the mechanism that returns > "credits" for the "hard limit" can become an additional burden for > implementations, depending on how that is done. A "loose" credit check > and/or update can significantly reduce that burden. As covered earlier on the list, I disagree that this is an unreasonable burden on the iWARP implementation. The only thing required is a simple credit count for each DDP stream endpoint However, Pat Thaler's post did raise legitimate scenarios where maintaining such a count would be an imposition on the local ULP. While I do not believe they represent a typical strategy, they are valid. So my suggestion is that the drafts move in the direction of being permissive as to the *mechanism* by which the limit is enforced. The security draft should note that use of a "soft" mechanism requires the local ULP to account for that softness and have sufficient slack to absorb the delay. I'll point out that a "soft" credit check combined with sequential ordering can require absorbing a *lot* of slack. But if a given local interface *can* work, then there is no reason not to allow it. Security drafts should provide guidance on how to evaluate the impact of a specific interface without enumerating them. Focusing on the two questions of *when* an invalid MSN is detected and when a buffer is bound to an MSN provides this type of analysis tool that should be applicable to any local interface. The goal should remain that accepting an untagged message with an invalid MSN should never put handling of a valid untagged message at risk -- especially when the valid untagged message is part of a different session. (A "session" for these purposes is a set of DDP Streams that serve the exact same set of masters. That is, if a fault on Stream X impairs Stream Y, there is no party that has a complaint because at the application layer Stream X and Stream Y do not represent distinct interests. Generally, a session is a single stream.) But I believe that goal can be stated without prescribing *specific* methods of achieving it. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 22 10:25:35 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA15342 for ; Tue, 22 Jul 2003 10:25:35 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ey4j-0000WT-MP for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 10:25:09 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6MEP9Mk001965 for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 10:25:09 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ey4j-0000Vc-Gx for rddp-web-archive@optimus.ietf.org; Tue, 22 Jul 2003 10:25:09 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA15311 for ; Tue, 22 Jul 2003 10:25:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ey4g-0004xC-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 10:25:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ey4b-0004x9-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 10:25:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ey4c-0000Uj-6n; Tue, 22 Jul 2003 10:25:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ey3x-0000SH-D1 for rddp@optimus.ietf.org; Tue, 22 Jul 2003 10:24:21 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA15267 for ; Tue, 22 Jul 2003 10:24:16 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ey3u-0004wU-00 for rddp@ietf.org; Tue, 22 Jul 2003 10:24:18 -0400 Received: from ztxmail03.ztx.compaq.com ([161.114.1.207]) by ietf-mx with esmtp (Exim 4.12) id 19ey3U-0004vw-00 for rddp@ietf.org; Tue, 22 Jul 2003 10:23:52 -0400 Received: from cceexg12.americas.cpqcorp.net (cceexg12.americas.cpqcorp.net [16.110.250.124]) by ztxmail03.ztx.compaq.com (Postfix) with ESMTP id D2711CE21 for ; Tue, 22 Jul 2003 09:22:46 -0500 (CDT) Received: from cceexc18.americas.cpqcorp.net ([16.81.1.4]) by cceexg12.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.6673); Tue, 22 Jul 2003 09:17:28 -0500 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 Content-Class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Date: Tue, 22 Jul 2003 09:17:12 -0500 Message-ID: <4D027986353D1341ADA4F2A4F8A170762AB1E937@cceexc18.americas.cpqcorp.net> Thread-Topic: MPA issues and next steps Thread-Index: AcNQW/DiCtBpabfyTCGiEisI7DmJbg== From: "Culley, Paul" To: X-OriginalArrivalTime: 22 Jul 2003 14:17:28.0606 (UTC) FILETIME=[FB5F03E0:01C3505B] Content-Transfer-Encoding: quoted-printable Subject: [rddp] MPA issues and next steps Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable As David has covered the workgroup issues around CRCs and Markers I think it is time to address some other issues. While we were short on time to discuss these things in Vienna, I did hear some opinions from people on what we need to do to complete the MPA "connection setup" process. 1) MPA will now require a "Startup Key" exchange prior to sending any FPDUs. 2) In the current draft, the "Startup Key" consists of a fixed ID, CRC flag, Marker Flag, and Revision. 3) It was proposed to add an "Active/Passive" indication to allow MPA to check that ULPs were starting MPA correctly (no Active/Active or Passive/Passive starts). 4) It was proposed to add a "Private Data" field and associated length field to the key. This would primarily be used when MPA starts up directly as TCP enters "established" state. 4a) The "private data" must be delivered on reception to the ULP, after which the ULP provides the parameters to enter full operations (QP, PD, initial buffers etc.), or to terminate the connection. This implies an MPA "mode" which can send and wait for the reception of the key, as well as appropriate interfaces to deal with the private data. 5) We need to show all the connection startup possibilities in the MPA draft, and discuss rules/issues with each: 5a) Startup immediately following TCP established with NO private data; MPA key exchange, followed by full operation. 5b) Startup immediately following TCP established with private data; MPA key/private data exchange, followed by full operation. 5c) Startup following prior streaming mode data exchanges; quiesce stream data, MPA key/private data exchange, followed by full operation. There were some opinions that said that the "Startup Key" and "Private data" responsibility could be left up to some ULP above the MPA layer. In this case the draft would only require that these be sent and received, but not say that MPA was itself responsible. I don't think we need to say this directly, but can simply say that an endpoint that is compliant with MPA must send and receive the key and private data. We can leave it to the vendors to describe what parts of the spec are executed by the hardware/device driver, and what parts are left to other layers. I suggest that to clarify the intent, that we provide an -EXAMPLE- ULP interface, that presumes that all of the MPA functionality is present in a single layer, similar to the function calls shown in the RDDP architecture draft. If, at some point, the WG decides to make a "verbs" document a WG item, then we can argue over the details of this interface in there. Any comments? Paul R. Culley HP Senior Fellow 281-514-5543 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 22 12:24:36 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA18844 for ; Tue, 22 Jul 2003 12:24:36 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ezvt-0005rz-Vf for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 12:24:10 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6MGO9Ae022563 for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 12:24:09 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ezvt-0005rq-Sp for rddp-web-archive@optimus.ietf.org; Tue, 22 Jul 2003 12:24:09 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA18823 for ; Tue, 22 Jul 2003 12:24:05 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ezvp-0005pd-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 12:24:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ezvk-0005pa-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 12:24:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ezvk-0005qA-Lh; Tue, 22 Jul 2003 12:24:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ezvP-0005pt-9o for rddp@optimus.ietf.org; Tue, 22 Jul 2003 12:23:39 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA18817 for ; Tue, 22 Jul 2003 12:23:34 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ezvN-0005pI-00 for rddp@ietf.org; Tue, 22 Jul 2003 12:23:37 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19ezvD-0005p3-00 for rddp@ietf.org; Tue, 22 Jul 2003 12:23:27 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id EC155245324; Tue, 22 Jul 2003 12:22:18 -0400 (EDT) Date: Tue, 22 Jul 2003 11:22:08 -0500 Subject: Re: [rddp] MPA issues and next steps Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: To: "Culley, Paul" From: Caitlin Bestler In-Reply-To: <4D027986353D1341ADA4F2A4F8A170762AB1E937@cceexc18.americas.cpqcorp.net> Message-Id: Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Tuesday, July 22, 2003, at 09:17 AM, Culley, Paul wrote: > > > 3) It was proposed to add an "Active/Passive" indication to allow MPA > to > check that ULPs were starting MPA correctly (no Active/Active or > Passive/Passive starts). In my experience Active/Active startup is most widely deployed in protocol manuals that have to spend several pages dealing with the corner cases on the states that are only required by active/active. Neither the DAT or IT-API provide any support for active/active. I've already figured out how to deal with it for the equivalent SCTP startup, but would prefer to simply delete it. With any luck your request and this observation will be met with thunderous silence, confirming that nobody has any real interest in active/active startup. I am curious though, exactly how a passive/passive startup would start. > 4) It was proposed to add a "Private Data" field and associated length > field to the key. This would primarily be used when MPA starts up > directly as TCP enters "established" state. > > 4a) The "private data" must be delivered on reception to the ULP, > after which the ULP provides the parameters to enter full operations > (QP, PD, initial buffers etc.), or to terminate the connection. This > implies an MPA "mode" which can send and wait for the reception of the > key, as well as appropriate interfaces to deal with the private data. > I believe this would be a valuable addition to the protocol. However, the procedures proposed in section 13 of the most recent draft are more than adequate. The key is to allow something *below* the ULP to support transport neutral stream initiation. I do not believe it is necessary for the entire algorithm to be implemented in the two adaptation layers. > 5) We need to show all the connection startup possibilities in the MPA > draft, and discuss rules/issues with each: > > 5a) Startup immediately following TCP established with NO private > data; MPA key exchange, followed by full operation. > > 5b) Startup immediately following TCP established with private data; > MPA key/private data exchange, followed by full operation. > > 5c) Startup following prior streaming mode data exchanges; quiesce > stream data, MPA key/private data exchange, followed by full operation. > > There were some opinions that said that the "Startup Key" and "Private > data" responsibility could be left up to some ULP above the MPA layer. > In this case the draft would only require that these be sent and > received, but not say that MPA was itself responsible. I don't think > we > need to say this directly, but can simply say that an endpoint that is > compliant with MPA must send and receive the key and private data. We > can leave it to the vendors to describe what parts of the spec are > executed by the hardware/device driver, and what parts are left to > other > layers. > > I suggest that to clarify the intent, that we provide an -EXAMPLE- ULP > interface, that presumes that all of the MPA functionality is present > in > a single layer, similar to the function calls shown in the RDDP > architecture draft. If, at some point, the WG decides to make a > "verbs" > document a WG item, then we can argue over the details of this > interface > in there. > I'd be happy to draft these as sections of the applicability draft, covering both MPA/TCP and SCTP in the same discussion. That would allow each LLP-specific draft to focus on what the implementers of that specific adaptation must do. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 22 19:56:15 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01196 for ; Tue, 22 Jul 2003 19:56:15 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f6yw-00066n-Pr for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 19:55:47 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6MNtkho023481 for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 19:55:46 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f6yw-00066Y-I0 for rddp-web-archive@optimus.ietf.org; Tue, 22 Jul 2003 19:55:46 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01155 for ; Tue, 22 Jul 2003 19:54:10 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19f6xP-0000Lq-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 19:54:11 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19f6xJ-0000Lm-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 19:54:05 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f6xE-00060T-Ev; Tue, 22 Jul 2003 19:54:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f6wR-0005z4-TI for rddp@optimus.ietf.org; Tue, 22 Jul 2003 19:53:12 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01147 for ; Tue, 22 Jul 2003 19:53:09 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19f6wQ-0000Lf-00 for rddp@ietf.org; Tue, 22 Jul 2003 19:53:10 -0400 Received: from mail4.microsoft.com ([131.107.3.122]) by ietf-mx with esmtp (Exim 4.12) id 19f6wD-0000KZ-00 for rddp@ietf.org; Tue, 22 Jul 2003 19:52:58 -0400 Received: from inet-vrs-04.redmond.corp.microsoft.com ([157.54.8.149]) by mail4.microsoft.com with Microsoft SMTPSVC(5.0.2195.6713); Tue, 22 Jul 2003 16:49:34 -0700 Received: from 157.54.8.23 by inet-vrs-04.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Tue, 22 Jul 2003 16:49:33 -0700 Received: from RED-IMC-04.redmond.corp.microsoft.com ([157.54.2.168]) by inet-hub-01.redmond.corp.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Tue, 22 Jul 2003 16:49:42 -0700 Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by RED-IMC-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Tue, 22 Jul 2003 16:49:26 -0700 Received: from WIN-MSG-10.wingroup.windeploy.ntdev.microsoft.com ([157.54.12.81]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Tue, 22 Jul 2003 16:49:29 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="------------InterScan_NT_MIME_Boundary" Subject: RE: [rddp] security doc - Definition of Trust Date: Tue, 22 Jul 2003 16:49:31 -0700 Message-ID: Thread-Topic: [rddp] security doc - Definition of Trust Thread-Index: AcNP7l6FNxbTF26OTh27cTiTOqNeSQAu45Nw From: "Jim Pinkerton" To: "Caitlin Bestler" Cc: X-OriginalArrivalTime: 22 Jul 2003 23:49:29.0123 (UTC) FILETIME=[E3FE4F30:01C350AB] Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multi-part message in MIME format. --------------InterScan_NT_MIME_Boundary Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C350AB.E60C0C1A" ------_=_NextPart_001_01C350AB.E60C0C1A Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable =20 You statement: =20 > For example, RDMAP/DDP could be implemented as a user-mode library > running on top of a traditional stack. If it only exposed the=20 > applications > own memory then there is no need for a "Privileged Resource Manager". > The looser rule, that the local interface grant the RNIC Engine no > authority that the caller itself did not have, is met by simply > running within the process. =20 Is exactly where you need a resource manager. There are fixed resources available, presumably (no-one has invented an infinite memory computer yet...) For example, on many systems random user can launch any user-mode code they wish. Thus anyone running on the system can attempt to be greedy and grab as much RNIC resources as possible, thereby making a DOS attack possible against other user mode clients, so long as they get the resources first. You could certainly argue you don't care about this attack - but it doesn't mean you shouldn't have a clean way of modeling the attack and showing those that care about this type of security issue how to address the attack. =20 =20 On another issue, I don't understand your statement of =20 " The descriptions of the Privileged Resource Manager presume a multi-process Unix/Windows style OS dealing with an RNIC Engine implemented primarily in hardware and/or fully enabled (i.e., kernel) memory space. That's certainly 99.99 whatever % of the world, but it isn't everything." =20 Where in the document does it make that statement? The document specifically defines it as: =20 "Privileged Resource Manager - the component responsible for managing and allocating resources associated with the RNIC Engine. The Resource Manager does not send or receive data." =20 It makes no statement about whether an operating system is present, whether an RNIC is in hardware or software, whether the resource manager is a procedure or a separate process, etc. It was meant to be as generic as possible. If we simply clarify the text, does that help? Maybe something like: =20 "Privileged Resource Manager - the component responsible for managing and allocating resources associated with the RNIC Engine. The Resource Manager does not send or receive data. Note that whether the Resource Manager is an independent component, part of the RNIC, or part of the application is implementation dependent. If a specific implementation does not wish to address security issues resolved by the Resource Manager, there may in fact be no resource manager at all." =20 =20 =20 Jim =20 =20 =20 -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com]=20 Sent: Monday, July 21, 2003 6:13 PM To: Jim Pinkerton Cc: rddp@ietf.org Subject: Re: [rddp] security doc - Definition of Trust =20 =20 On Monday, July 21, 2003, at 07:58 PM, Jim Pinkerton wrote: =20 > On the idea of removing the privileged resource manager, I guess I'm=20 > unclear on the advantage of it. It's just a reference model to help=20 > clarify issues. There are no MUSTs anywhere. Using the text as you=20 > describe basically collapses the functionality into the RNIC. To me=20 > one of the intentions of the paper was to specifically slice up the=20 > security issues into more detail to better explain the issues. How you > implement it is your business though. =20 I'm trying to be inclusive of all possible implementations. =20 For example, RDMAP/DDP could be implemented as a user-mode library running on top of a traditional stack. If it only exposed the=20 applications own memory then there is no need for a "Privileged Resource Manager". The looser rule, that the local interface grant the RNIC Engine no authority that the caller itself did not have, is met by simply running within the process. =20 At the opposite extreme, I have worked with system buses where the driver was allowed to directly DMA into a user map. (Not that anyone has done anything like that *recently*, but ideas have a way of cycling back around. Who knows about 3 years from now?). =20 So I'd like to recognize that there are multiple ways of ensuring that an RNIC Engine does not violate the host system's security rules. The descriptions of the Privileged Resource Manager presume a multi-process Unix/Windows style OS dealing with an RNIC Engine implemented primarily in hardware and/or fully enabled (i.e., kernel) memory space. That's certainly 99.99 whatever % of the world, but it isn't everything. =20 Perhaps if the principle were established that some form of access validation would be required, and that it would likely be required to start from a single root authorizer. The Privileged Resource Manager described could then be given as an example. =20 =20 Caitlin Bestler - cait@asomi.com - http://asomi.com/ =20 =20 ------_=_NextPart_001_01C350AB.E60C0C1A Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable

 

You statement:

 

> For example, RDMAP/DDP could be implemented as a user-mode = library

> running on top of a traditional stack. If it only exposed = the

> applications

> own memory then there is no need for a "Privileged = Resource Manager".

> The looser rule, that the local interface grant the RNIC = Engine no

> authority that the caller itself did not have, is met by = simply

> running within the process.

 

Is exactly where you need a resource manager. There are fixed = resources available, presumably (no-one has invented an infinite memory computer = yet…) For example, on many systems random user can launch any user-mode code = they wish. Thus anyone running on the system can attempt to be greedy and = grab as much RNIC resources as possible, thereby making a DOS attack possible = against other user mode clients, so long as they get the resources first. You = could certainly argue you don’t care about this attack – but it = doesn’t mean you shouldn’t have a clean way of modeling the attack and = showing those that care about this type of security issue how to address the = attack.

 

 

On another issue, I don't understand your statement = of

 

" The descriptions of the Privileged = Resource Manager

presume a multi-process Unix/Windows style OS = dealing with

an RNIC Engine implemented primarily in = hardware and/or

fully enabled (i.e., kernel) memory space. = That's certainly

99.99 whatever % of the world, but it isn't everything."

 

Where in the document does it make that statement? The document = specifically defines it as:

 

"Privileged Resource Manager – the component responsible for managing and allocating resources associated = with the RNIC Engine. The Resource Manager does not send or receive = data."

 

It makes no statement about whether an operating system is = present, whether an RNIC is in hardware or software, whether the resource manager = is a procedure or a separate process, etc. It was meant to be as generic as possible. If we simply clarify the text, does that help? Maybe something = like:

 

"Privileged Resource Manager – the component = responsible for managing and allocating resources associated with the RNIC Engine. The = Resource Manager does not send or receive data. Note that whether the Resource = Manager is an independent component, part of the RNIC, or part of the = application is implementation dependent. If a specific implementation does not wish to = address security issues resolved by the Resource Manager, there may in fact be = no resource manager at all."

 

 

 

Jim

 

 

 

-----Original Message-----
From: Caitlin Bestler [mailto:cait@asomi.com]
Sent: Monday, July 21, 2003 6:13 PM
To: Jim Pinkerton
Cc: rddp@ietf.org
Subject: Re: [rddp] security doc - Definition of Trust

 

 

On Monday, July 21, 2003, at 07:58 PM, Jim Pinkerton = wrote:

 

> On the idea of removing the privileged resource manager, I = guess I'm

> unclear on the advantage of it. It's just a reference model = to help

> clarify issues. There are no MUSTs anywhere. Using the text = as you

> describe basically collapses the functionality into the = RNIC. To me

> one of the intentions of the paper was to specifically = slice up the

> security issues into more detail to better explain the = issues. How you

> implement it is your business though.

 

I'm trying to be inclusive of all possible = implementations.

 

For example, RDMAP/DDP could be implemented as a user-mode = library

running on top of a traditional stack. If it only exposed the =

applications

own memory then there is no need for a "Privileged Resource Manager".

The looser rule, that the local interface grant the RNIC Engine = no

authority that the caller itself did not have, is met by = simply

running within the process.

 

At the opposite extreme, I have worked with system buses where = the

driver was allowed to directly DMA into a user map. (Not that = anyone

has done anything like that *recently*, but ideas have a way = of

cycling back around. Who knows about 3 years from = now?).

 

So I'd like to recognize that there are multiple ways of = ensuring

that an RNIC Engine does not violate the host system's = security

rules. The descriptions of the Privileged Resource = Manager

presume a multi-process Unix/Windows style OS dealing = with

an RNIC Engine implemented primarily in hardware = and/or

fully enabled (i.e., kernel) memory space. That's = certainly

99.99 whatever % of the world, but it isn't = everything.

 

Perhaps if the principle were established that some form = of

access validation would be required, and that it would = likely

be required to start from a single root authorizer. The = Privileged

Resource Manager described could then be given as an = example.

 

 

Caitlin Bestler - cait@asomi.com - = http://asomi.com/

 

 

------_=_NextPart_001_01C350AB.E60C0C1A-- --------------InterScan_NT_MIME_Boundary-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 22 19:59:36 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01295 for ; Tue, 22 Jul 2003 19:59:36 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f72B-0006Ep-Io for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 19:59:08 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6MNx7vE023973 for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 19:59:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f72B-0006Ea-Ej for rddp-web-archive@optimus.ietf.org; Tue, 22 Jul 2003 19:59:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01290 for ; Tue, 22 Jul 2003 19:59:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19f729-0000Nt-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 19:59:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19f723-0000Nq-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 19:58:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f724-0006Cy-My; Tue, 22 Jul 2003 19:59:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f71B-0006CN-5O for rddp@optimus.ietf.org; Tue, 22 Jul 2003 19:58:05 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01262 for ; Tue, 22 Jul 2003 19:58:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19f719-0000NC-00 for rddp@ietf.org; Tue, 22 Jul 2003 19:58:03 -0400 Received: from mail2.microsoft.com ([131.107.3.124]) by ietf-mx with esmtp (Exim 4.12) id 19f70y-0000MU-00 for rddp@ietf.org; Tue, 22 Jul 2003 19:57:52 -0400 Received: from mail6.microsoft.com ([157.54.6.196]) by mail2.microsoft.com with Microsoft SMTPSVC(5.0.2195.6713); Tue, 22 Jul 2003 16:56:01 -0700 Received: from inet-vrs-06.redmond.corp.microsoft.com ([157.54.6.181]) by mail6.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Tue, 22 Jul 2003 16:55:59 -0700 Received: from 157.54.8.109 by inet-vrs-06.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Tue, 22 Jul 2003 16:55:59 -0700 Received: from red-imc-02.redmond.corp.microsoft.com ([157.54.9.107]) by inet-hub-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Tue, 22 Jul 2003 16:55:44 -0700 Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Tue, 22 Jul 2003 16:55:38 -0700 Received: from WIN-MSG-10.wingroup.windeploy.ntdev.microsoft.com ([157.54.12.81]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Tue, 22 Jul 2003 16:55:54 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [rddp] WG chair on buffer exhaustion Date: Tue, 22 Jul 2003 16:55:55 -0700 Message-ID: Thread-Topic: [rddp] WG chair on buffer exhaustion Thread-Index: AcNQWbcIwHXrmDKWTmCRHDNdWRRF3gAUvceg From: "Jim Pinkerton" To: "Caitlin Bestler" , "Culley, Paul" Cc: , X-OriginalArrivalTime: 22 Jul 2003 23:55:54.0790 (UTC) FILETIME=[C9DE6C60:01C350AC] Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable I agree that the security draft should focus on requirements, not on implementation details. I'll try to draft something up to be reviewed by this group that doesn't choose one implementation over the other. Jim -----Original Message----- From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org] On Behalf Of Caitlin Bestler Sent: Tuesday, July 22, 2003 6:59 AM To: Culley, Paul Cc: Black_David@emc.com; rddp@ietf.org Subject: Re: [rddp] WG chair on buffer exhaustion On Tuesday, July 22, 2003, at 08:29 AM, Culley, Paul wrote: > As pointed out earlier in this thread, the mechanism that returns > "credits" for the "hard limit" can become an additional burden for > implementations, depending on how that is done. A "loose" credit check > and/or update can significantly reduce that burden. As covered earlier on the list, I disagree that this is an unreasonable burden on the iWARP implementation. The only thing required is a simple credit count for each DDP stream endpoint However, Pat Thaler's post did raise legitimate scenarios where=20 maintaining such a count would be an imposition on the local ULP. While I do not=20 believe they represent a typical strategy, they are valid. So my suggestion is that the drafts move in the direction of being=20 permissive as to the *mechanism* by which the limit is enforced. The security=20 draft should note that use of a "soft" mechanism requires the local ULP to account=20 for that softness and have sufficient slack to absorb the delay. I'll point out=20 that a "soft" credit check combined with sequential ordering can require=20 absorbing a *lot* of slack. But if a given local interface *can* work, then there is no reason not=20 to allow it. Security drafts should provide guidance on how to evaluate the impact=20 of a specific interface without enumerating them. Focusing on the two questions of=20 *when* an invalid MSN is detected and when a buffer is bound to an MSN provides this type=20 of analysis tool that should be applicable to any local interface. The goal should remain that accepting an untagged message with an=20 invalid MSN should never put handling of a valid untagged message at risk -- especially=20 when the valid untagged message is part of a different session. (A "session" for these=20 purposes is a set of DDP Streams that serve the exact same set of masters. That is,=20 if a fault on Stream X impairs Stream Y, there is no party that has a complaint=20 because at the application layer Stream X and Stream Y do not represent distinct=20 interests. Generally, a session is a single stream.) But I believe that goal can be stated without prescribing *specific*=20 methods of achieving it. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 22 20:15:35 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01577 for ; Tue, 22 Jul 2003 20:15:35 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f7Hg-0006h9-2i for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 20:15:08 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6N0F8Tv025730 for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 20:15:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f7Hf-0006gv-W7 for rddp-web-archive@optimus.ietf.org; Tue, 22 Jul 2003 20:15:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01552 for ; Tue, 22 Jul 2003 20:15:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19f7Hd-0000Sp-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 20:15:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19f7HY-0000Sm-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 20:15:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f7HZ-0006f9-5R; Tue, 22 Jul 2003 20:15:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f7Gg-0006dl-Q8 for rddp@optimus.ietf.org; Tue, 22 Jul 2003 20:14:06 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA01541 for ; Tue, 22 Jul 2003 20:14:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19f7Ge-0000S7-00 for rddp@ietf.org; Tue, 22 Jul 2003 20:14:04 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19f7GT-0000RU-00 for rddp@ietf.org; Tue, 22 Jul 2003 20:13:54 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id EA75F24541F; Tue, 22 Jul 2003 20:12:40 -0400 (EDT) Date: Tue, 22 Jul 2003 19:12:30 -0500 Subject: Re: [rddp] security doc - Definition of Trust Content-Type: text/plain; charset=WINDOWS-1252; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: To: "Jim Pinkerton" From: Caitlin Bestler In-Reply-To: Message-Id: <59A56A2A-BCA2-11D7-AE83-003065D48EE0@asomi.com> Content-Transfer-Encoding: quoted-printable X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable On Tuesday, July 22, 2003, at 06:49 PM, Jim Pinkerton wrote: > =A0 > > Your statement: > > =A0 > > > For example, RDMAP/DDP could be implemented as a user-mode library > > > running on top of a traditional stack. If it only exposed the > > > applications > > > own memory then there is no need for a "Privileged Resource = Manager". > > > The looser rule, that the local interface grant the RNIC Engine no > > > authority that the caller itself did not have, is met by simply > > > running within the process. > > =A0 > > Is exactly where you need a resource manager. There are fixed=20 > resources available, presumably (no-one has invented an infinite=20 > memory computer yet=85) For example, on many systems random user can=20= > launch any user-mode code they wish. Thus anyone running on the system=20= > can attempt to be greedy and grab as much RNIC resources as possible,=20= > thereby making a DOS attack possible against other user mode clients,=20= > so long as they get the resources first. You could certainly argue you=20= > don=92t care about this attack =96 but it doesn=92t mean you shouldn=92t= have=20 > a clean way of modeling the attack and showing those that care about=20= > this type of security issue how to address the attack. > > =A0 The "RNIC" has limited resources, because it is software running *within* the context of the user process. Typically a Privileged Resource Manager is required because the RNIC is hardware that executes in a context where the OS cannot veto its actions. On most platforms, hardware must be trusted. That is different than the common "OS bypass" model, where a user-mode library interfaces more or less directly with the RNIC. You are quite correct that under such a model, a Privileged Resource Manager (or some equivalent) is required to validate the resource authority of each of these libraries. This is mostly a language point. The security draft should deal with all implementations. It is of course totally appropriate to point out that typically the RNIC Engine must operate with system-wide privileges. And that requires the existence of a Privileged Resource Manager. But you *can* implement the protocol without assuming system- wide privileges. It may even be possible to have a Java Applet client. It would be silly, but it would be a legal client. > > =A0 > > On another issue, I don't understand your statement of > > =A0 > > " The descriptions of the Privileged Resource Manager > > presume a multi-process Unix/Windows style OS dealing with > > an RNIC Engine implemented primarily in hardware and/or > > fully enabled (i.e., kernel) memory space. That's certainly > > 99.99 whatever % of the world, but it isn't everything." > > =A0 > > Where in the document does it make that statement? The document=20 > specifically defines it as: > > =A0 > > "Privileged Resource Manager =96 the component responsible for = managing=20 > and allocating resources associated with the RNIC Engine. The Resource=20= > Manager does not send or receive data." > > =A0 > > It makes no statement about whether an operating system is present,=20 > whether an RNIC is in hardware or software, whether the resource=20 > manager is a procedure or a separate process, etc. It was meant to be=20= > as generic as possible. The concept that access to resources needs privileged control assumes that the RNIC is operating in a context that is at least partially something other than normal user software. > If we simply clarify the text, does that help? Maybe something like: > > =A0 > > "Privileged Resource Manager =96 the component responsible for = managing=20 > and allocating resources associated with the RNIC Engine. The Resource=20= > Manager does not send or receive data. Note that whether the Resource=20= > Manager is an independent component, part of the RNIC, or part of the=20= > application is implementation dependent. If a specific implementation=20= > does not wish to address security issues resolved by the Resource=20 > Manager, there may in fact be no resource manager at all." > > > The real distinction is not whether an implementation *wants* to address security issues, it is a question of whether the implementation needed to be placed outside of the normal OS-specific security mechanisms (i.e., it is running in hardware and/or the kernel). Does the RNIC engine require privileges that a simple user process would not have, such as accessing memory in multiple user maps? Virtually all implementations *will* be in hardware and/or kernel based. When the RNIC engine requires *special* privileges then a Resource Manager is indeed a requirement. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 22 20:42:34 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA02032 for ; Tue, 22 Jul 2003 20:42:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f7hn-0007cN-Uk for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 20:42:07 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6N0g70w029278 for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 20:42:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f7hn-0007c9-OZ for rddp-web-archive@optimus.ietf.org; Tue, 22 Jul 2003 20:42:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA02019 for ; Tue, 22 Jul 2003 20:42:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19f7hl-0000ab-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 20:42:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19f7hf-0000aY-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 20:41:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f7hh-0007Xn-6G; Tue, 22 Jul 2003 20:42:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f7hB-0007XN-0s for rddp@optimus.ietf.org; Tue, 22 Jul 2003 20:41:29 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA02008 for ; Tue, 22 Jul 2003 20:41:25 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19f7h8-0000aL-00 for rddp@ietf.org; Tue, 22 Jul 2003 20:41:26 -0400 Received: from mail2.microsoft.com ([131.107.3.124]) by ietf-mx with esmtp (Exim 4.12) id 19f7gy-0000aF-00 for rddp@ietf.org; Tue, 22 Jul 2003 20:41:16 -0400 Received: from INET-VRS-02.redmond.corp.microsoft.com ([157.54.8.110]) by mail2.microsoft.com with Microsoft SMTPSVC(5.0.2195.6713); Tue, 22 Jul 2003 17:40:41 -0700 Received: from 157.54.5.25 by INET-VRS-02.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Tue, 22 Jul 2003 17:40:28 -0700 Received: from red-imc-01.redmond.corp.microsoft.com ([157.54.9.102]) by inet-hub-03.redmond.corp.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Tue, 22 Jul 2003 17:40:45 -0700 Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-01.redmond.corp.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Tue, 22 Jul 2003 17:40:41 -0700 Received: from WIN-MSG-10.wingroup.windeploy.ntdev.microsoft.com ([157.54.12.81]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Tue, 22 Jul 2003 17:40:39 -0700 X-MimeOLE: Produced By Microsoft Exchange V6.5.6944.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Subject: RE: [rddp] security doc - Definition of Trust Date: Tue, 22 Jul 2003 17:40:40 -0700 Message-ID: Thread-Topic: [rddp] security doc - Definition of Trust Thread-Index: AcNQryDCvpzNh0d0TvGZrNNAuzEydQAAdrdw From: "Jim Pinkerton" To: "Caitlin Bestler" Cc: X-OriginalArrivalTime: 23 Jul 2003 00:40:39.0837 (UTC) FILETIME=[0A4814D0:01C350B3] Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable I think your point at the end is a good one: > Virtually all implementations *will* be in hardware and/or kernel = based.=20 > When the RNIC engine requires *special* privileges then a Resource = Manager > is indeed a requirement. Thus using language that maps well to that environment while not = restricting in any way the implementation seems appropriate. That was = the approach intended by the authors. If you note, in the text we tried = to be extremely careful to define terms. I think part of the issue here = is you aren't applying the strict definitions in the paper. You're = adding your own baggage as to what "privileged" means, and whether that = applies to hardware or software implementations. From the ID, there is never a mention of user-mode verses kernel mode, = other than in an example (or at least it was not intended, and we'll fix = it if there is!). Same for what is done in hardware vs. software - there = is never a distinction made - both are implementation issues. We dealt = only with "Privileged" and "Non-Privileged". In the context of an = application, the contrast was defined as: "The Privileged Application is partially trusted. It is assumed that the = Privileged Application will not intentionally attack the system (e.g., = it is a kernel application), although it may be greedy for resources." "A Non-Privileged Application's capabilities are a logical sub-set of = the Privileged Application's. It is assumed by the local host = infrastructure that a Non-Privileged Application is Untrusted. All = Non-Privileged Application interactions with the RNIC Engine that could = affect other applications need to be done through a Trusted intermediary = that can verify the Non-Privileged Application requests." Further in the specification, the Privileged Application Data Transfer = Interface is defined to be a superset of the Non-Privileged Application = Data Transfer Interface: "Privileged Data Transfer Interface - A superset of the functionality = provided by the Non-Privileged Data Transfer Interface. The application = is allowed to directly manipulate RNIC Engine mapping resources to map = an STag to an application data buffer." Not sure if this helps or not, but I wanted to make sure you understood = the intention of the authors. We tried very hard to be generic - I think = the biggest issue I see here is you believe using the word "privileged" = implies something, but the document does not appear to support your = assumptions? Jim -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com]=20 Sent: Tuesday, July 22, 2003 5:13 PM To: Jim Pinkerton Cc: rddp@ietf.org Subject: Re: [rddp] security doc - Definition of Trust On Tuesday, July 22, 2003, at 06:49 PM, Jim Pinkerton wrote: > =A0 > > Your statement: > > =A0 > > > For example, RDMAP/DDP could be implemented as a user-mode library > > > running on top of a traditional stack. If it only exposed the > > > applications > > > own memory then there is no need for a "Privileged Resource = Manager". > > > The looser rule, that the local interface grant the RNIC Engine no > > > authority that the caller itself did not have, is met by simply > > > running within the process. > > =A0 > > Is exactly where you need a resource manager. There are fixed=20 > resources available, presumably (no-one has invented an infinite=20 > memory computer yet...) For example, on many systems random user can=20 > launch any user-mode code they wish. Thus anyone running on the system = > can attempt to be greedy and grab as much RNIC resources as possible,=20 > thereby making a DOS attack possible against other user mode clients,=20 > so long as they get the resources first. You could certainly argue you = > don't care about this attack - but it doesn't mean you shouldn't have=20 > a clean way of modeling the attack and showing those that care about=20 > this type of security issue how to address the attack. > > =A0 The "RNIC" has limited resources, because it is software running *within* the context of the user process. Typically a Privileged Resource Manager is required because the RNIC is hardware that executes in a context where the OS cannot veto its actions. On most platforms, hardware must be trusted. That is different than the common "OS bypass" model, where a user-mode library interfaces more or less directly with the RNIC. You are quite correct that under such a model, a Privileged Resource Manager (or some equivalent) is required to validate the resource authority of each of these libraries. This is mostly a language point. The security draft should deal with all implementations. It is of course totally appropriate to point out that typically the RNIC Engine must operate with system-wide privileges. And that requires the existence of a Privileged Resource Manager. But you *can* implement the protocol without assuming system- wide privileges. It may even be possible to have a Java Applet client. It would be silly, but it would be a legal client. > > =A0 > > On another issue, I don't understand your statement of > > =A0 > > " The descriptions of the Privileged Resource Manager > > presume a multi-process Unix/Windows style OS dealing with > > an RNIC Engine implemented primarily in hardware and/or > > fully enabled (i.e., kernel) memory space. That's certainly > > 99.99 whatever % of the world, but it isn't everything." > > =A0 > > Where in the document does it make that statement? The document=20 > specifically defines it as: > > =A0 > > "Privileged Resource Manager - the component responsible for managing=20 > and allocating resources associated with the RNIC Engine. The Resource = > Manager does not send or receive data." > > =A0 > > It makes no statement about whether an operating system is present,=20 > whether an RNIC is in hardware or software, whether the resource=20 > manager is a procedure or a separate process, etc. It was meant to be=20 > as generic as possible. The concept that access to resources needs privileged control assumes that the RNIC is operating in a context that is at least partially something other than normal user software. > If we simply clarify the text, does that help? Maybe something like: > > =A0 > > "Privileged Resource Manager - the component responsible for managing=20 > and allocating resources associated with the RNIC Engine. The Resource = > Manager does not send or receive data. Note that whether the Resource=20 > Manager is an independent component, part of the RNIC, or part of the=20 > application is implementation dependent. If a specific implementation=20 > does not wish to address security issues resolved by the Resource=20 > Manager, there may in fact be no resource manager at all." > > > The real distinction is not whether an implementation *wants* to address security issues, it is a question of whether the implementation needed to be placed outside of the normal OS-specific security mechanisms (i.e., it is running in hardware and/or the kernel). Does the RNIC engine require privileges that a simple user process would not have, such as accessing memory in multiple user maps? Virtually all implementations *will* be in hardware and/or kernel based. When the RNIC engine requires *special* privileges then a Resource Manager is indeed a requirement. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 22 21:04:34 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA02492 for ; Tue, 22 Jul 2003 21:04:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f836-0008CB-2n for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 21:04:08 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6N1485d031497 for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 21:04:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f835-0008Bw-Vh for rddp-web-archive@optimus.ietf.org; Tue, 22 Jul 2003 21:04:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA02489 for ; Tue, 22 Jul 2003 21:04:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19f833-0000iW-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 21:04:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19f82x-0000iT-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 21:03:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f82z-0008AF-7E; Tue, 22 Jul 2003 21:04:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19f82v-0008A4-3q for rddp@optimus.ietf.org; Tue, 22 Jul 2003 21:03:57 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA02486 for ; Tue, 22 Jul 2003 21:03:53 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19f82s-0000iQ-00 for rddp@ietf.org; Tue, 22 Jul 2003 21:03:54 -0400 Received: from mxic2.corp.emc.com ([128.221.31.40]) by ietf-mx with esmtp (Exim 4.12) id 19f82i-0000iH-00 for rddp@ietf.org; Tue, 22 Jul 2003 21:03:44 -0400 Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Tue, 22 Jul 2003 21:02:52 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0C246759@corpmx14.us.dg.com> To: rddp@ietf.org Subject: RE: [rddp] WG chair on buffer exhaustion Date: Tue, 22 Jul 2003 20:52:34 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Paul, > Do I read into this that you believe that DDP implementations > MUST implement a per stream hard limit on untagged messages > ("termination is a reasonable response")? Pending further work on the security draft, I'm not certain about either "MUST implement", or the precise definition of "hard limit". I would prefer to allow work on the security draft to determine the requirements to protect shared resources rather than preemptively decide exactly what is necessary as WG chair now. > If you do, the next question is how fast? Is it ok to delay > the stream termination under some conditions, until some > "extra" untagged messages have arrived and used buffers? > This might occur if the mechanism for detecting the hard > limit exception depended on counting completions in software, > for example, or if it depended on software to respond to an exception. If this class of mechanism is needed, that sort of delay may be ok. The determining factors will be the level and nature of protection of the shared untagged receive resources from exhaustion that is necessary, and "necessary" will probably be in the sense of "available to the ULP" (i.e., "MUST implement", not "MUST use"). > As pointed out earlier in this thread, the mechanism that > returns "credits" for the "hard limit" can become an > additional burden for implementations, depending on how that > is done. A "loose" credit check and/or update can > significantly reduce that burden. Yes, and this was also clear from the discussion in Vienna. > It is not clear how much guidance the DDP security specs need > to provide on this issue, other than to say something to the > effect of: "Here is an issue, make sure that your > implementation, or upper interface and ULP can deal with it." > Seems like getting too detailed may provide unnecessary > restrictions on implementations. Assuming that we do have a requirement to protect shared receive resources for untagged messages in this fashion, the suggested text is a bit too loose - ULPs will need to rely on somewhat tighter functional behavior promises than "make sure that your ... can deal with it", as the resulting engineering for the worst case will waste resources. Thanks, --David > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Monday, July 21, 2003 2:03 PM > > To: rddp@ietf.org > > Subject: [rddp] WG chair on buffer exhaustion > > > > > > The heated debate on whether lack of availability of a buffer for an > > untagged message causes immediate termination of the DDP stream on > > which it arrived caused some additional email behind the scenes in > > addition to what's been on the list. > > > > With my WG chair hat on, here is what I believe to be the correct > > resolution of this issue: > > > > The request to do something other than immediately terminate a stream > > when a message shows up that does not have a pre-queued untagged > > buffer to receive it does not require that DDP tell TCP not to ACK the > > message; an implementation could choose to do that based on > > inter-layer implementation optimizations, but had better do so very rarely. > > Alternatively, a DDP implementation could choose to have some > > "hidden" resources in its back pocket that might involve a > > less efficient receive path (e.g., receive the errant message > > to a hidden kernel buffer and then copy to an application > > buffer when it becomes available) > > - again, this would need to be a rare occurrence for obvious > > reasons. This sort of approach is more robust than having TCP > > not ACK, as it avoids propagating a host problem into the > > network and puts the performance penalty for the > > implementation problem (should have had a buffer available, > > but didn't) squarely on the system that had the problem. > > > > I would prefer to see the "hidden [kernel] buffer" approach > > described as an example rather than the "TCP doesn't ACK" approach. > > Among the other problems with "TCP doesn't ACK" is that it creates a > > nasty head-of-line blocking problem that affects all traffic on the > > connection, not just the untagged traffic that is facing the > > resource shortage. > > > > OTOH, if the ULP has set a hard limit on the number of untagged > > messages that can come in from a specific stream, and that stream > > exceeds the limit, this is evidence that the counterparty isn't > > following the rules, and termination of the stream is a reasonable > > response. In the particular scenario that John raises below, the > > hard limit would need to be set high enough to accommodate > > error/special/ emergency conditions, and statistical/shared buffer > > provisioning would probably be used to avoid having to provision > > buffers for simultaneous occurrence of error/special/emergency > > conditions on all inbound streams. > > > > Thanks, > > --David > > ---------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 176 South St., Hopkinton, MA 01748 > > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > ---------------------------------------------------- > > > > > > > > > >On Wednesday, July 16, 2003, hufferd@us.ibm.com wrote: > > > > > > > >>Again, I think that Mallikarjun stated things correctly. There > > > >>are important applications which can only correctly predict the > > > >>number of buffers needed in non error/special/emergency > > > >>conditions. When error/special/emergency conditions occur, the > > > >>number and urgency of the handling is problematical. > > > > _______________________________________________ > > rddp mailing list > > rddp@ietf.org > > https://www1.ietf.org/mailman/listinfo/rddp > > > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 22 23:43:35 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA04568 for ; Tue, 22 Jul 2003 23:43:35 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fAWy-0004CZ-SM for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 23:43:08 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6N3h8T5016145 for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 23:43:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fAWy-0004CK-NW for rddp-web-archive@optimus.ietf.org; Tue, 22 Jul 2003 23:43:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA04559 for ; Tue, 22 Jul 2003 23:43:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19fAWw-0001MS-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 23:43:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19fAWr-0001MP-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 23:43:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fAWr-0004BU-BV; Tue, 22 Jul 2003 23:43:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fAWi-0004B6-Nv for rddp@optimus.ietf.org; Tue, 22 Jul 2003 23:42:52 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA04553 for ; Tue, 22 Jul 2003 23:42:48 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19fAWg-0001MG-00 for rddp@ietf.org; Tue, 22 Jul 2003 23:42:50 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19fAWW-0001M2-00 for rddp@ietf.org; Tue, 22 Jul 2003 23:42:40 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 0D646245A83 for ; Tue, 22 Jul 2003 23:41:55 -0400 (EDT) Date: Tue, 22 Jul 2003 22:41:44 -0500 Subject: Re: [rddp] WG chair on buffer exhaustion Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) From: Caitlin Bestler To: rddp@ietf.org Content-Transfer-Encoding: 7bit In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0C246759@corpmx14.us.dg.com> Message-Id: <9498C864-BCBF-11D7-AE83-003065D48EE0@asomi.com> X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Tuesday, July 22, 2003, at 07:52 PM, Black_David@emc.com wrote: > If this class of mechanism is needed, that sort of delay may > be ok. The determining factors will be the level and nature > of protection of the shared untagged receive resources from > exhaustion that is necessary, and "necessary" will probably > be in the sense of "available to the ULP" (i.e., "MUST implement", > not "MUST use"). Which suggests that the obvious set of options to consider are: - MUST support hard or soft limits. MAY support both. - MUST support hard limits. MAY support soft. - MUST support soft limits. MAY support hard. - MUST support hard and soft limits (but not necessarily both limits on any single stream). My initial hunch is to favor the first or the last option, with a slight preference for the last one. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 22 23:55:35 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA04891 for ; Tue, 22 Jul 2003 23:55:35 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fAia-0004lQ-US for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 23:55:09 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6N3t8x8018306 for rddp-archive@odin.ietf.org; Tue, 22 Jul 2003 23:55:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fAia-0004lB-Q9 for rddp-web-archive@optimus.ietf.org; Tue, 22 Jul 2003 23:55:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA04883 for ; Tue, 22 Jul 2003 23:55:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19fAiY-0001QX-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 23:55:06 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19fAiS-0001QU-00 for rddp-web-archive@ietf.org; Tue, 22 Jul 2003 23:55:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fAiT-0004gU-LS; Tue, 22 Jul 2003 23:55:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fAhw-0004g0-MY for rddp@optimus.ietf.org; Tue, 22 Jul 2003 23:54:28 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA04877 for ; Tue, 22 Jul 2003 23:54:24 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19fAhu-0001QL-00 for rddp@ietf.org; Tue, 22 Jul 2003 23:54:26 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19fAhj-0001Pu-00 for rddp@ietf.org; Tue, 22 Jul 2003 23:54:16 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id E5DBF245923; Tue, 22 Jul 2003 23:36:01 -0400 (EDT) Date: Tue, 22 Jul 2003 22:35:51 -0500 Subject: Re: [rddp] security doc - Definition of Trust Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: rddp@ietf.org To: "Jim Pinkerton" From: Caitlin Bestler In-Reply-To: Message-Id: Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Tuesday, July 22, 2003, at 07:40 PM, Jim Pinkerton wrote: > Not sure if this helps or not, but I wanted to make sure you > understood the intention of the authors. We tried very hard to be > generic - I think the biggest issue I see here is you believe using > the word "privileged" implies something, but the document does not > appear to support your assumptions? I stand corrected. The appropriate dodge words were already in place. I missed them, but as was already pointed out the distinctions weren't meant to be important to most readers anyway. So they didn't really need to be emphasized, just there somewhere. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 23 16:19:00 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA01172 for ; Wed, 23 Jul 2003 16:19:00 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fQ4G-0004gY-UI for rddp-archive@odin.ietf.org; Wed, 23 Jul 2003 16:18:33 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6NKIWkG018006 for rddp-archive@odin.ietf.org; Wed, 23 Jul 2003 16:18:32 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fQ4G-0004gL-RO for rddp-web-archive@optimus.ietf.org; Wed, 23 Jul 2003 16:18:32 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA01149 for ; Wed, 23 Jul 2003 16:18:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fQ3l-0004eZ-Lo; Wed, 23 Jul 2003 16:18:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fQ3H-0004cF-7Y for rddp@optimus.ietf.org; Wed, 23 Jul 2003 16:17:31 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA01100 for ; Wed, 23 Jul 2003 16:17:28 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19fQ3F-0000T1-00 for rddp@ietf.org; Wed, 23 Jul 2003 16:17:29 -0400 Received: from fmr02.intel.com ([192.55.52.25] helo=caduceus.fm.intel.com) by ietf-mx with esmtp (Exim 4.12) id 19fQ34-0000Sq-00 for rddp@ietf.org; Wed, 23 Jul 2003 16:17:18 -0400 Received: from talaria.fm.intel.com (talaria.fm.intel.com [10.1.192.39]) by caduceus.fm.intel.com (8.11.6p2/8.11.6/d: outer.mc,v 1.66 2003/05/22 21:17:36 rfjohns1 Exp $) with ESMTP id h6NK8CX17554 for ; Wed, 23 Jul 2003 20:08:13 GMT Received: from fmsmsxvs041.fm.intel.com (fmsmsxvs041.fm.intel.com [132.233.42.126]) by talaria.fm.intel.com (8.11.6p2/8.11.6/d: inner.mc,v 1.35 2003/05/22 21:18:01 rfjohns1 Exp $) with SMTP id h6NKH3m12744 for ; Wed, 23 Jul 2003 20:17:03 GMT Received: from fmsmsx331.amr.corp.intel.com ([132.233.42.135]) by fmsmsxvs041.fm.intel.com (NAVGW 2.5.2.11) with SMTP id M2003072313124824291 ; Wed, 23 Jul 2003 13:12:50 -0700 Received: from fmsmsx402.amr.corp.intel.com ([132.233.42.200]) by fmsmsx331.amr.corp.intel.com with Microsoft SMTPSVC(5.0.2195.5329); Wed, 23 Jul 2003 13:16:39 -0700 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 Subject: RE: [rddp] WG chair on buffer exhaustion Date: Wed, 23 Jul 2003 13:16:38 -0700 Message-ID: <6315617889C99D4BA7C14687DEC8DB4E02BC7518@fmsmsx402.fm.intel.com> Thread-Topic: [rddp] WG chair on buffer exhaustion Thread-Index: AcNQzIvf5nwHOzPBSiymHtRM6zJq+QAiCl0A From: "Shah, Hemal" To: "Caitlin Bestler" , X-OriginalArrivalTime: 23 Jul 2003 20:16:39.0058 (UTC) FILETIME=[52DAC720:01C35157] Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable See my comments below. Hemal -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com]=20 Sent: Tuesday, July 22, 2003 10:42 PM To: rddp@ietf.org Subject: Re: [rddp] WG chair on buffer exhaustion On Tuesday, July 22, 2003, at 07:52 PM, Black_David@emc.com wrote: > If this class of mechanism is needed, that sort of delay may > be ok. The determining factors will be the level and nature > of protection of the shared untagged receive resources from > exhaustion that is necessary, and "necessary" will probably > be in the sense of "available to the ULP" (i.e., "MUST implement", > not "MUST use"). Which suggests that the obvious set of options to consider are: - MUST support hard or soft limits. MAY support both. - MUST support hard limits. MAY support soft. - MUST support soft limits. MAY support hard. - MUST support hard and soft limits (but not necessarily both limits on any single stream). [hvs] I think for all of the above options it makes sense to have only one limit (hard or soft) on any single stream. Your last option seems to suggest that for the first three options, one can have hard, or soft, or both limits specified per DDP stream. [hvs] My initial hunch is to favor the first or the last option, with a slight preference for the last one. [hvs] I am in favor of the option below (close to your option 3): RDMAP/DDP implementation MUST support soft limit. RDMAP/DDP implementation MAY support hard limit. Only one limit can be specified per RDMAP/DDP stream. My second preference will be your option 4. [hvs}=20 Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 23 16:23:34 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA01365 for ; Wed, 23 Jul 2003 16:23:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fQ8h-0004v2-AR for rddp-archive@odin.ietf.org; Wed, 23 Jul 2003 16:23:07 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6NKN7lp018902 for rddp-archive@odin.ietf.org; Wed, 23 Jul 2003 16:23:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fQ8h-0004un-6e for rddp-web-archive@optimus.ietf.org; Wed, 23 Jul 2003 16:23:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA01331 for ; Wed, 23 Jul 2003 16:23:04 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19fQ8f-0000UZ-00 for rddp-web-archive@ietf.org; Wed, 23 Jul 2003 16:23:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19fQ8Z-0000UW-00 for rddp-web-archive@ietf.org; Wed, 23 Jul 2003 16:22:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fQ8a-0004tB-M0; Wed, 23 Jul 2003 16:23:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fQ85-0004sN-5H for rddp@optimus.ietf.org; Wed, 23 Jul 2003 16:22:29 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA01295 for ; Wed, 23 Jul 2003 16:22:26 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19fQ83-0000U7-00 for rddp@ietf.org; Wed, 23 Jul 2003 16:22:27 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19fQ7s-0000U3-00 for rddp@ietf.org; Wed, 23 Jul 2003 16:22:16 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id B4578245C0B; Wed, 23 Jul 2003 16:21:34 -0400 (EDT) Date: Wed, 23 Jul 2003 15:21:27 -0500 Subject: Re: [rddp] WG chair on buffer exhaustion Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: To: "Shah, Hemal" From: Caitlin Bestler In-Reply-To: <6315617889C99D4BA7C14687DEC8DB4E02BC7518@fmsmsx402.fm.intel.com> Message-Id: <3CC08D0F-BD4B-11D7-B498-003065D48EE0@asomi.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Wednesday, July 23, 2003, at 03:16 PM, Shah, Hemal wrote: > [hvs] I think for all of the above options it makes sense to have only > one limit (hard or soft) on any single stream. Your last option seems > to > suggest that for the first three options, one can have hard, or soft, > or > both limits specified per DDP stream. [hvs] Agreed. With any of the options, the implementation should only be expected to support one limit on any stream. Checking a soft-limit and *then* a hard-limit would be too complex for a MUST or even a SHOULD. -------------------- Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 23 19:38:34 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA07915 for ; Wed, 23 Jul 2003 19:38:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fTBP-0006Mr-Hn for rddp-archive@odin.ietf.org; Wed, 23 Jul 2003 19:38:07 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6NNc7lR024471 for rddp-archive@odin.ietf.org; Wed, 23 Jul 2003 19:38:07 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fTBP-0006Mc-Dp for rddp-web-archive@optimus.ietf.org; Wed, 23 Jul 2003 19:38:07 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA07900 for ; Wed, 23 Jul 2003 19:38:03 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19fTBN-0001kA-00 for rddp-web-archive@ietf.org; Wed, 23 Jul 2003 19:38:05 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19fTBI-0001k7-00 for rddp-web-archive@ietf.org; Wed, 23 Jul 2003 19:38:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fTBI-0006M9-Ma; Wed, 23 Jul 2003 19:38:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19fTAX-0006A3-Kv for rddp@optimus.ietf.org; Wed, 23 Jul 2003 19:37:13 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA07896 for ; Wed, 23 Jul 2003 19:37:09 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19fTAV-0001jm-00 for rddp@ietf.org; Wed, 23 Jul 2003 19:37:11 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19fTAL-0001jI-00 for rddp@ietf.org; Wed, 23 Jul 2003 19:37:01 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 325222459F5 for ; Wed, 23 Jul 2003 19:34:09 -0400 (EDT) Date: Wed, 23 Jul 2003 18:34:02 -0500 Mime-Version: 1.0 (Apple Message framework v552) Content-Type: text/plain; charset=US-ASCII; format=flowed From: Caitlin Bestler To: rddp@ietf.org Content-Transfer-Encoding: 7bit Message-Id: <2423596E-BD66-11D7-B498-003065D48EE0@asomi.com> X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Subject: [rddp] Robust handling of untagged messages Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit A careful reading of the current DDP draft may be compatible with the current consensus of allowing an implementation to be robust in the face of a buffer shortage. Indeed it maybe possible to clarify that the desired behavior is a valid implementation without making any changes to the draft. While many required interactions between DDP and the LLP are outlined, there is nothing that implies that this is a description of the complete interface. Therefore, the LLP layer could invoke an interface that performs placement *only if possible*. The possible results of this interface would indicate whether the placement had been accepted or not. The DDP layer could reject a placement which cannot be handled at the current moment, essentially asking the LLP layer to stall the packet somehow or another. The full semantics of "placement" and "delivery" would then be intact, whenever the DDP layer accepted the placement. ------------------ Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Jul 25 09:46:39 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA04872 for ; Fri, 25 Jul 2003 09:46:39 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19g2ti-0004Gl-5z for rddp-archive@odin.ietf.org; Fri, 25 Jul 2003 09:46:14 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6PDkEkF016405 for rddp-archive@odin.ietf.org; Fri, 25 Jul 2003 09:46:14 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19g2ti-0004GW-1f for rddp-web-archive@optimus.ietf.org; Fri, 25 Jul 2003 09:46:14 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA04866 for ; Fri, 25 Jul 2003 09:46:08 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19g2tg-0000Ng-00 for rddp-web-archive@ietf.org; Fri, 25 Jul 2003 09:46:12 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19g2ta-0000Na-00 for rddp-web-archive@ietf.org; Fri, 25 Jul 2003 09:46:06 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19g2tV-0004Ec-SU; Fri, 25 Jul 2003 09:46:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19g2sn-0004B1-6M for rddp@optimus.ietf.org; Fri, 25 Jul 2003 09:45:17 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA04829 for ; Fri, 25 Jul 2003 09:45:12 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19g2sl-0000N8-00 for rddp@ietf.org; Fri, 25 Jul 2003 09:45:15 -0400 Received: from mxic2.corp.emc.com ([128.221.31.40]) by ietf-mx with esmtp (Exim 4.12) id 19g2sa-0000MO-00 for rddp@ietf.org; Fri, 25 Jul 2003 09:45:04 -0400 Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Fri, 25 Jul 2003 09:44:09 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0C606403@corpmx14.us.dg.com> To: rddp@ietf.org Date: Fri, 25 Jul 2003 09:33:43 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] Architecture draft resolution Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Everyone, Speaking as WG chair ... Based on what I described in Vienna, here is the direction for resolution of the shared resource issue that is the last open issue on the architecture draft: In addition to restoring ddp_post_receive, the earlier definition of post_receive needs to be expanded to allow for under-provisioning. That is: a) a single buffer may be enabled on multiple streams. b) under-provisioning creates potential problems with resource exhaustion. Local interfaces should provide some method for handling this. (Thunderous silence in the architecture draft on what these mechanisms might be). A -03 version of the architecture draft will be prepared in accordance with this and forwarded to the ADs in the near future Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Jul 25 18:52:28 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA00375 for ; Fri, 25 Jul 2003 18:52:28 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19gBPw-0006Yv-Vj for rddp-archive@odin.ietf.org; Fri, 25 Jul 2003 18:52:05 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6PMq4Mn025222 for rddp-archive@odin.ietf.org; Fri, 25 Jul 2003 18:52:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19gBPw-0006Yj-R5 for rddp-web-archive@optimus.ietf.org; Fri, 25 Jul 2003 18:52:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA00367 for ; Fri, 25 Jul 2003 18:51:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19gBPt-0005zR-00 for rddp-web-archive@ietf.org; Fri, 25 Jul 2003 18:52:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19gBPt-0005zO-00 for rddp-web-archive@ietf.org; Fri, 25 Jul 2003 18:52:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19gBPt-0006YK-Ke; Fri, 25 Jul 2003 18:52:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19gBPi-0006Y6-Gi for rddp@optimus.ietf.org; Fri, 25 Jul 2003 18:51:50 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA00361 for ; Fri, 25 Jul 2003 18:51:43 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19gBPf-0005zD-00 for rddp@ietf.org; Fri, 25 Jul 2003 18:51:47 -0400 Received: from mxic2.corp.emc.com ([128.221.31.40]) by ietf-mx with esmtp (Exim 4.12) id 19gBPe-0005z2-00 for rddp@ietf.org; Fri, 25 Jul 2003 18:51:46 -0400 Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Fri, 25 Jul 2003 18:51:16 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0C246775@corpmx14.us.dg.com> To: rddp@ietf.org Date: Fri, 25 Jul 2003 18:40:50 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] RDDP WG Vienna minutes - DRAFT Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Here are the draft minutes. Comments to me and/or the list, please. Based on the results of the Vienna meeting, I've asked the secretariat to confer official RDDP WG draft status on the current MPA and Security drafts (they'll get linked to the WG's charter page on the web), but I expect the draft authors to submit revised versions with draft-ietf-rddp-... names soon. Thanks, --David RDDP WG minutes - Vienna, Austria, July 2003 - DRAFT -------------------------------------------- Thanks to Tom Talpey for taking the initial minutes. Letters in [square brackets] indicate first letter in name of corresponding presentation in proceedings. Tuesday, July 15 5-6pm ---------------------- David Black (WG chair) introduces agenda changes, no bashing, agenda accepted [A] Review of WG draft intentions and status - see presentation [B] + RDDP Applicability statement, Lode Coene [C] Review of document content - draft considered by authors to be in good shape, waiting for comments from WG. David Black requests that TCP mapping issues (drawbacks of TCP by comparison to SCTP) be included in the applicability statement. + SCTP Mapping draft [D] On its third revision. A DDP Stream establishment issue has been identified, can a protection domain assignment be deferred or no? Needs documentation at a minimum. A draft revision under author review concerns single-message exchange for "private data", compatible with current MPA draft, but this is part of the open startup issues in the MPA draft. No others issues are currently identified, speak up now... New version expected in next few weeks. + Shared receive queues vs Shared buffer pool [D] David Black presented this issue from the mailing list on behalf of Caitlin Bestler, who was unable to travel to Vienna. Shared receive queues are proposed in draft-hilland-verbs-00. Their presence/absence has impact on other drafts: Architecture document should it define a "post receive"? Security issues The problem: need to underprovision receive buffers, but how to control per-receive queue quotas? Two proposals: Shared receive queue (draft-hilland) Shared buffer pool (email on list) The proposals are described and a lengthy discussion ensues. David Black suggests trying the following tack: need to resolve robustness to DOS attack on shared buffers. Tag this as the only open portion of this issue and look at countermeasures in security draft. Examine offline and continue this issue on Thursday. A question is asked about why this issue is holding up the architecture draft, which is *not* an interface specification. David Black answers that the architecture draft goes into sufficient detail on pseudo-interfaces that some discussion of resource sharing is necessary there. + RDMAP/DDP Security draft - Jim Pinkerton [E] Presents the new security draft for first time. Draft is focused on wire protocol issues, in a generic way with out referencing verbs, etc. Sketches components necessary to implement security over RDMA. - privileged resource manager, controlling the RNIC itself as well as two classes of application - priv and non-priv applications. Identifies resources to be analyzed (and controlled): connection context, data buffers, translation tables, STags, completion queues, plus RDMA Read request queue. Describes the "dimensions of trust": local resource sharing, local trust and remote trust. There are 8 combinations of trust, 4 are addressed in the paper. The 4 were chosen as most applicable to IETF areas of interest, due to brevity and also lack of relevant applications which might fall into the other 4. Enumerates tools for countermeasures: protection domain, limiting STag scope, (4 others) David Black requests that for Thursday's session, protocols should decide which "trust bucket" each lands in, determines if bucket is addressed in draft, and if so are the issues accurate? Thursday, July 17, 3:30-5:30pm ------------------------------ + Architecture draft resolution ddp_post_receive goes back in Allow sharing of buffers for untagged receives Work out how to avoid tacking limit per stream/socket problem in arch draft; problem will be addressed in security draft In ddp draft, ok to not fail immediately on shared overdraw question from floor - have we addressed the issues raised by VJ delay-bandwidth results? Enough buffers to allow the flow to use all available bw? A (David Black): This is an issue for receivers to appropriately provision receive buffers. Consider adding a statement to this effect to the architecture draft. + Return to Security document discussion, Jim Pinkerton. [This meeting time conflicts with SAAG meeting so security experts may not be in attendance.] Current open issues summarized: - IPSec section is currently a placeholder, and will be filled in (Bernard Aboba to help out) - Also, summary section at end still in progress. - From reflector, elaborate some issues in "trust model". - Multiple Protection Domains - Scope of STags - Other resources to add to security discussion (async event queue, PD, etc) - STag disable/enable by non-privileged app. Question from floor - we need an analysis of issues introduced by RDMA into existing protocols, e.g. NFSv4. When transfers are done purely by RDMA they may not be protected by existing security. A (David Black): This may indeed require additional security coverage such as IPsec which may be enabled by existing CCM NFSv4 proposal. It as also noted that the integrity protection is present in NFSv4, RPCSEC_GSS integrity, can verify that RDMA transfers actually occur as promised, while IPsec can be used to enforce the authorizations themselves. Two different issues, important to keep separate. Question - should performance analysis be included in our analysis? Especially where IPSec overhead is involved. A (David Black): First of all, let's complete the security analysis. At that point, the specific performance implications may be worth analyzing. But not before. Next major revision of this document should become an official WG draft. No objections. + Status of TCP mapping issues WG chair summarizes marker issues resolved to date: Sender alignment Marker usage, intervals, optionality CRC use (5+ issues) Further discussion to go into applicability draft Question from floor - have we worked out when it's ok not to use MPA CRC yet? Prefers to have it always present for simplicity. A (David Black): Rough consensus is for CRC to be optional. Discussion of when it may be disabled and what ULP designers/implementers should assume goes into the applicability statement draft. Last(?) CRC Issue - one CRC or two? Case for two - faster placement, fast header validation Case for one - tractable, simpler, doesn't abuse layering Statement from floor - there are more reasons to have two CRCs: DDP Routers don't have to rebuild data crc Don't have to wait for whole segment to begin processing header Large amount of controversy whether it's valid to rewrite DDP headers. WG Chair declares that rough consensus exists for one CRC; while there are advantages to two, they don't outweigh increased complexity. Further discussion will be on the list. + MPA discussion - Paul Culley Changes to most recent draft recapped: optional, receiver-specified markers optional, mutually agreed CRC MPA streaming mode "key" validation mechanism at connect new informational sections Authors' additional proposals no "alignment" of market to TCP sequence #, as alignment would violate layering CRC field always present, not filed in or checked when not used - simplicity no private data in MPA (carried only in TCP mode), quiescing stream required at MPA init MPA 128-bit start key, carrying a fixed MPA signal + CRC/marker bits, etc. passive/active rules specified Question - what happens if both "active"? Needs to be addressed Resolved - Start Key should at a minimum assert Active vs. Passive. Discussion of connection models - immediate RDMA enabled, RDMA enabled midstream, RDMA midstream with private data (blob follows MPA key). This whole area of how to initiate a connection, including private data, is open and needs to go to list Chair proposes that this document is ready to go to official WG draft for TCP mapping of RDDP (with significant issues to be worked out of course). No objections. Question: Is the MPA draft as informational or standards track? A (WG Chair) Informational: per ADs from recent rechartering. Can be reconsidered when draft is complete and ready for publication as an RFC. Q: Didn't acceptance of Start Key mechanism enable standards track status? A (WG Chair): No, refusal to accept Start Key mechanism would have resulted in TCP mapping work being removed from RDDP WG charter. NOTE: WG Chair has subsequently verified w/ADs that standards track status for the TCP mapping (MPA) can be reconsidered when draft is complete; there's a strong desire to see exactly what is being proposed and understand its effects on TCP before approving standards track status. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Jul 28 15:31:34 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA18011 for ; Mon, 28 Jul 2003 15:31:34 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hDi6-0004EA-VG for rddp-archive@odin.ietf.org; Mon, 28 Jul 2003 15:31:07 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6SJV65E016248 for rddp-archive@odin.ietf.org; Mon, 28 Jul 2003 15:31:06 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hDi6-0004Dz-Rw for rddp-web-archive@optimus.ietf.org; Mon, 28 Jul 2003 15:31:06 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA18005 for ; Mon, 28 Jul 2003 15:31:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19hDi4-0006zl-00 for rddp-web-archive@ietf.org; Mon, 28 Jul 2003 15:31:04 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19hDi3-0006zh-00 for rddp-web-archive@ietf.org; Mon, 28 Jul 2003 15:31:03 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hDi1-0004DV-9p; Mon, 28 Jul 2003 15:31:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hDhm-0004Cx-1p for rddp@optimus.ietf.org; Mon, 28 Jul 2003 15:30:46 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA17993 for ; Mon, 28 Jul 2003 15:30:42 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19hDhk-0006zb-00 for rddp@ietf.org; Mon, 28 Jul 2003 15:30:44 -0400 Received: from palrel13.hp.com ([156.153.255.238]) by ietf-mx with esmtp (Exim 4.12) id 19hDhj-0006zX-00 for rddp@ietf.org; Mon, 28 Jul 2003 15:30:43 -0400 Received: from rosemail.rose.hp.com (rosemail.rose.hp.com [15.96.64.26]) by palrel13.hp.com (Postfix) with ESMTP id EEE061C019FE for ; Mon, 28 Jul 2003 12:30:37 -0700 (PDT) Received: from swathi (pal1nai164024.nsr.hp.com [15.244.164.24]) by rosemail.rose.hp.com with SMTP (8.7.1/8.7.3 SMKit7.02) id MAA28800 for ; Mon, 28 Jul 2003 12:30:35 -0700 (PDT) Message-ID: <005e01c3553e$b8845740$18a4f40f@rose.hp.com> From: "Mallikarjun C." To: Date: Mon, 28 Jul 2003 12:30:31 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Content-Transfer-Encoding: 7bit Subject: [rddp] Fw: draft-chadalapaka-iwarp-da-00.txt,.pdf Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit This may be of interest to this mailing list as well. Apologies if you receive multiple copies. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Hewlett-Packard MS 5668 Roseville CA 95747 cbm@rose.hp.com ----- Original Message ----- From: "Mallikarjun C." To: Sent: Friday, July 25, 2003 5:24 PM Subject: draft-chadalapaka-iwarp-da-00.txt,.pdf > All: > > I submitted an Internet Draft with the subject name on behalf of a team of > authors. The I-D announcement, including the abstract, is below. This > draft can be obtained from: > > http://www.ietf.org/internet-drafts/draft-chadalapaka-iwarp-da-00.pdf > http://www.ietf.org/internet-drafts/draft-chadalapaka-iwarp-da-00.txt > > "Datamover Architecture for iSCSI" (DA) describes an architectural > model for iSCSI so that new "Datamover protocols" - such as iSER, > see the separate announcement forwarded to the list by Mike Ko - > can be designed based on this architecture. Recommended reading > is DA, followed by iSER. The relevant related specifications (RDMAP, > DDP and MPA) are currently being defined by the RDDP WG. > > Review and comments are welcome. > -- > Mallikarjun > > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions > Hewlett-Packard MS 5668 > Roseville CA 95747 > cbm@rose.hp.com > > > > I-D ACTION:draft-chadalapaka-iwarp-da-00.txt,.pdf > > > > To: IETF-Announce: ; > Subject: I-D ACTION:draft-chadalapaka-iwarp-da-00.txt,.pdf > From: Internet-Drafts@ietf.org > Date: Fri, 25 Jul 2003 15:27:02 -0400 > Cc: ips@ece.cmu.edu > Reply-to: Internet-Drafts@ietf.org > Sender: owner-ietf-announce@ietf.org > > > > A New Internet-Draft is available from the on-line Internet-Drafts directories. > > > Title : Datamover Architecture for iSCSI (DA) > Author(s) : M. Chadalapaka et al. > Filename : draft-chadalapaka-iwarp-da-00.txt,.pdf > Pages : 60 > Date : 2003-7-25 > > iSCSI is a SCSI transport protocol that maps the SCSI family of > application protocols onto TCP/IP. The Datamover Architecture for > iSCSI (DA) defines an abstract model in which the movement of data > between iSCSI end nodes is logically separated from the rest of the > iSCSI protocol in order to allow iSCSI to adapt to innovations > available in new IP transports. The new Datamover protocol provides > a reliable transport for all iSCSI PDUs, but actually moves > the data required for certain iSCSI PDUs without involving the > remote iSCSI layer itself. This document begins with an > introduction of a few new abstractions, defines a layered > architecture for iSCSI and Datamover protocols, and then models the > interactions within an iSCSI end node between the iSCSI layer and > the Datamover layer that happen in order to transparently perform > remote data movement within an IP fabric. It is intended that this > definition would help map iSCSI to generic RDMA-capable IP fabrics > in the future comprising TCP, SCTP, and possibly other underlying > network transport layers. > > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-chadalapaka-iwarp-da-00.txt > > To remove yourself from the IETF Announcement list, send a message to > ietf-announce-request with the word unsubscribe in the body of the message. > > Internet-Drafts are also available by anonymous FTP. Login with the username > "anonymous" and a password of your e-mail address. After logging in, > type "cd internet-drafts" and then > "get draft-chadalapaka-iwarp-da-00.txt". > > A list of Internet-Drafts directories can be found in > http://www.ietf.org/shadow.html > or ftp://ftp.ietf.org/ietf/1shadow-sites.txt > > > Internet-Drafts can also be obtained by e-mail. > > Send a message to: > mailserv@ietf.org. > In the body type: > "FILE /internet-drafts/draft-chadalapaka-iwarp-da-00.txt". > > NOTE: The mail server at ietf.org can return the document in > MIME-encoded form by using the "mpack" utility. To use this > feature, insert the command "ENCODING mime" before the "FILE" > command. To decode the response(s), you will need "munpack" or > a MIME-compliant mail reader. Different MIME-compliant mail readers > exhibit different behavior, especially when dealing with > "multipart" MIME messages (i.e. documents which have been split > up into multiple messages), so check your local documentation on > how to manipulate these messages. > > > Below is the data which will enable a MIME compliant mail reader > implementation to automatically retrieve the ASCII version of the > Internet-Draft. > > > > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 29 15:42:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA06868 for ; Tue, 29 Jul 2003 15:42:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19haMF-0001lI-9F for rddp-archive@odin.ietf.org; Tue, 29 Jul 2003 15:42:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6TJg3FT006768 for rddp-archive@odin.ietf.org; Tue, 29 Jul 2003 15:42:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19haMF-0001l5-5h for rddp-web-archive@optimus.ietf.org; Tue, 29 Jul 2003 15:42:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA06852 for ; Tue, 29 Jul 2003 15:41:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19haMD-0001I2-00 for rddp-web-archive@ietf.org; Tue, 29 Jul 2003 15:42:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19haMD-0001Hy-00 for rddp-web-archive@ietf.org; Tue, 29 Jul 2003 15:42:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19haMD-0001kh-5Y; Tue, 29 Jul 2003 15:42:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19haLV-0001kF-Lf for rddp@optimus.ietf.org; Tue, 29 Jul 2003 15:41:17 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA06843 for ; Tue, 29 Jul 2003 15:41:12 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19haLT-0001Ht-00 for rddp@ietf.org; Tue, 29 Jul 2003 15:41:15 -0400 Received: from atlrel7.hp.com ([156.153.255.213]) by ietf-mx with esmtp (Exim 4.12) id 19haLT-0001Hp-00 for rddp@ietf.org; Tue, 29 Jul 2003 15:41:15 -0400 Received: from rosemail.rose.hp.com (rosemail.rose.hp.com [15.96.64.26]) by atlrel7.hp.com (Postfix) with ESMTP id C79AA1C01EA0; Tue, 29 Jul 2003 15:41:13 -0400 (EDT) Received: from swathi (dhcp-roseor-bdj047.rose.hp.com [15.23.139.47]) by rosemail.rose.hp.com with SMTP (8.7.1/8.7.3 SMKit7.02) id MAA16108; Tue, 29 Jul 2003 12:41:12 -0700 (PDT) Message-ID: <012001c35609$599f4390$18a4f40f@rose.hp.com> From: "Mallikarjun C." To: , References: <277DD60FB639D511AC0400B0D068B71E0C246750@corpmx14.us.dg.com> Subject: Re: [rddp] WG chair on buffer exhaustion Date: Tue, 29 Jul 2003 12:39:43 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit David, Sorry it took me a while to respond to this discussion. Also, I will only have sporadic access to email for the rest of the week. I guess I generally agree with the direction you're suggesting. I had earlier suggested certain DDP text changes to address this issue. Based on your input, here's one more attempt to address the same. I hope this is acceptable to everyone since most seem to agree that the current DDP text is too restrictive/severe in this area. Regards. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Hewlett-Packard MS 5668 Roseville CA 95747 cbm@rose.hp.com The current relevant text is: section 3.2 (Protocol Overview), " Note, however, that the ULP needs to address flow control issues because if a DDP Message arrives for an Untagged Buffer without an associated receive ULP buffer, the DDP Message is dropped, the DDP Stream is disabled for reception, and an error is reported to the ULP at the Data Sink." and section 9.1 (Errors detected at the Data Sink). " 2. The QN and MSN have an associated buffer that allows Placement of the payload." My suggestion is to change the text along the following lines: Section 3.2 to: " Note, however, that the ULP needs to address flow control issues. Any ULP that runs on DDP SHOULD consider the lack of an associated receive buffer to place an inbound Untagged Message of the ULP as an exception." and section 9.1 to: " 2. The QN and MSN have an associated buffer that allows Placement of the payload. In order to gracefully deal with transient buffer exhaustion, some DDP implementations MAY choose to consider lack of an associated receive buffer to be not an error , such implementations MUST however ensure that the affected DDP Segment is eventually recovered in an implementation-specific (as an example, by relying on DDP private buffers) way that guarantees ULP-transparency. However, the ULP SHOULD NOT rely on the DDP layer gracefully handling this situation on a regular basis." ----- Original Message ----- From: To: Sent: Monday, July 21, 2003 12:02 PM Subject: [rddp] WG chair on buffer exhaustion > The heated debate on whether lack of availability of a buffer for > an untagged message causes immediate termination of the DDP stream > on which it arrived caused some additional email behind the scenes > in addition to what's been on the list. > > With my WG chair hat on, here is what I believe to be the correct > resolution of this issue: > > The request to do something other than immediately terminate > a stream when a message shows up that does not have a pre-queued > untagged buffer to receive it does not require that DDP tell > TCP not to ACK the message; an implementation could choose to > do that based on inter-layer implementation optimizations, > but had better do so very rarely. Alternatively, a DDP > implementation could choose to have some "hidden" resources > in its back pocket that might involve a less efficient receive > path (e.g., receive the errant message to a hidden kernel buffer > and then copy to an application buffer when it becomes available) > - again, this would need to be a rare occurrence for obvious reasons. > This sort of approach is more robust than having TCP not ACK, as > it avoids propagating a host problem into the network and puts the > performance penalty for the implementation problem (should > have had a buffer available, but didn't) squarely on the system > that had the problem. > > I would prefer to see the "hidden [kernel] buffer" approach > described as an example rather than the "TCP doesn't ACK" approach. > Among the other problems with "TCP doesn't ACK" is that it creates > a nasty head-of-line blocking problem that affects all traffic > on the connection, not just the untagged traffic that is facing > the resource shortage. > > OTOH, if the ULP has set a hard limit on the number of untagged > messages that can come in from a specific stream, and that stream > exceeds the limit, this is evidence that the counterparty isn't > following the rules, and termination of the stream is a reasonable > response. In the particular scenario that John raises below, the hard > limit would need to be set high enough to accommodate error/special/ > emergency conditions, and statistical/shared buffer provisioning would > probably be used to avoid having to provision buffers for simultaneous > occurrence of error/special/emergency conditions on all inbound streams. > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > black_david@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > > > > > > >On Wednesday, July 16, 2003, hufferd@us.ibm.com wrote: > > > > > >>Again, I think that Mallikarjun stated things correctly. There are > > >>important applications which can only correctly predict the number of > > >>buffers needed in non error/special/emergency conditions. When > > >>error/special/emergency conditions occur, the number and urgency of the > > >>handling is problematical. > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Tue Jul 29 16:53:27 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA08850 for ; Tue, 29 Jul 2003 16:53:26 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hbSw-00052s-84 for rddp-archive@odin.ietf.org; Tue, 29 Jul 2003 16:53:02 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6TKr2RJ019388 for rddp-archive@odin.ietf.org; Tue, 29 Jul 2003 16:53:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hbSw-00052c-4G for rddp-web-archive@optimus.ietf.org; Tue, 29 Jul 2003 16:53:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA08834 for ; Tue, 29 Jul 2003 16:52:56 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19hbSu-0001k3-00 for rddp-web-archive@ietf.org; Tue, 29 Jul 2003 16:53:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19hbSt-0001k0-00 for rddp-web-archive@ietf.org; Tue, 29 Jul 2003 16:52:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hbSu-00052H-Ha; Tue, 29 Jul 2003 16:53:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hbSI-0004xw-4p for rddp@optimus.ietf.org; Tue, 29 Jul 2003 16:52:22 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA08824 for ; Tue, 29 Jul 2003 16:52:16 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19hbSG-0001jr-00 for rddp@ietf.org; Tue, 29 Jul 2003 16:52:20 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19hbSF-0001jb-00 for rddp@ietf.org; Tue, 29 Jul 2003 16:52:19 -0400 Received: from 192.168.0.216 (208-41-143-203.client.dsl.net [208.41.143.203]) by thebe.your-site.com (Postfix) with ESMTP id 8DDC1245B48; Tue, 29 Jul 2003 16:51:58 -0400 (EDT) Date: Tue, 29 Jul 2003 15:51:49 -0500 From: Caitlin Bestler Subject: Re: [rddp] WG chair on buffer exhaustion To: "Mallikarjun C." Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <012001c35609$599f4390$18a4f40f@rose.hp.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit > My suggestion is to change the text along the following lines: > > Section 3.2 to: > " Note, however, that the ULP needs to address flow > control issues. Any ULP that runs on DDP SHOULD > consider the lack of an associated receive buffer to > place an inbound Untagged Message of the ULP as an > exception." > I would suggest the following section: Section TBD ULP Flow-control The ULP is responsible for providing flow control for untagged messages. The Data Source MUST know that it is authorized to send an untagged message before sending it. The Data Sink MUST know the maximum number of untagged messages that it has authorized for reception on a DDP Stream or set of streams. The method by which the Data Source and Data Sink ULP reach agreement on this limit is outside the scope of this specification. Possible methods include dynamically negotiated credits, session negotiated credits and fixed limits imposed by the ULP specification. The Data Sink SHOULD provide sufficient buffers for the untagged messages that it has authorized. However, it is not required to do so by pre-committing a buffer for each valid QN/MSN combination. However the local interface MUST allow the Data Sink ULP to pre-specify the buffer for each valid QN/MSN combination. Receipt of an untagged message that was not authorized by the Data Sink ULP SHOULD result in termination of the DDP Stream. The local interface may allow authorization of untagged messages for a specific stream and/or for a group of streams. Receipt of an untagged message that was authorized, but for which there is no available buffer, MAY result in termination of the DDP Stream. However, implementations MAY treat the absence of a buffer for a valid untaged message as a local fault. Implementation specific procedures to recover from such a fault MAY be used, especially if they avoid impacting the remote peer. I believe this specifically takes care of one iSER concern. The bulk of the allowed untagged messages have CmdSNs, and are flow controlled for a *set* of streams that belong to a single session. It would require that a limit be set for asynchronous messages, but would allow that to be a ULP constant rather than a dynamically negotiated value. It also allows for strict and immediate termination of untagged messages that are bound to simple Receive Queues or to the RDMA Read Engine. In the latter case, a "soft limit" approach would seem to be very inappropriate. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 30 12:08:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA06483 for ; Wed, 30 Jul 2003 12:08:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19htUg-0006uI-U9 for rddp-archive@odin.ietf.org; Wed, 30 Jul 2003 12:08:05 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6UG82Pc026546 for rddp-archive@odin.ietf.org; Wed, 30 Jul 2003 12:08:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19htUg-0006u5-RB for rddp-web-archive@optimus.ietf.org; Wed, 30 Jul 2003 12:08:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA06476 for ; Wed, 30 Jul 2003 12:07:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19htUf-0001Q7-00 for rddp-web-archive@ietf.org; Wed, 30 Jul 2003 12:08:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19htUf-0001Q3-00 for rddp-web-archive@ietf.org; Wed, 30 Jul 2003 12:08:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19htUe-0006tk-VI; Wed, 30 Jul 2003 12:08:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19htUc-0006tG-3R for rddp@optimus.ietf.org; Wed, 30 Jul 2003 12:07:58 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA06468 for ; Wed, 30 Jul 2003 12:07:52 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19htUa-0001Pv-00 for rddp@ietf.org; Wed, 30 Jul 2003 12:07:56 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19htUZ-0001Pe-00 for rddp@ietf.org; Wed, 30 Jul 2003 12:07:56 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id DAF5D245DBF; Wed, 30 Jul 2003 12:07:33 -0400 (EDT) Date: Wed, 30 Jul 2003 11:07:22 -0500 Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: , rddp@ietf.org To: "Mallikarjun C." From: Caitlin Bestler In-Reply-To: <016301c3563e$0bf013b0$18a4f40f@rose.hp.com> Message-Id: Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Subject: [rddp] Re: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Tuesday, July 29, 2003, at 08:58 PM, Mallikarjun C. wrote: > As Mike points out, the CmdSN-based flow control > in iSCSI is relevant here. Let me note that the design > team behind the current iSER draft considered this topic > in great detail, but I can now clearly see that the draft > unfortunately does not capture the design rationale very well. > It could be clearer. But even if it were clearer, it would not change the fact that it fails to provide ULP-level flow control for untagged messages. The requirement here is that the ULP provide *flow control* for untagged messages. Control means that the Data Source either has permission to send an untagged message, or it does not. There is an identical flow control issue for RDMA Reads. You either are allowed to send one or you are not. If you are allowed to send an untagged message, you have an expectation that the other side has the resources to handle it. Bugs, under-provisioning and hardware faults are all facts of life. So robust applications are prepared to deal with faults. But faults reflect a *failure*. If an untagged message is allowed, then the Data Source has ever reason to expect that the Data Sink will handle it properly. Failure to do so if a fault on the Data Sink. If an untagged message is not allowed, then the Data Source had no right to send it. It cannot complain if the Data Sink terminates the stream. In this case the Data Source is the one committing the fault. > iSCSI does not provide a PDU-level positive flow control > but instead relies on the CmdSN feature, from which most > of the iSCSI (what DA/iSER call as the) "control-type" PDU traffic > can be precisely estimated (note that only control-type PDUs > are candidates for Send Messages and thus relevant to this > discussion). However, it turns out that there are certain > opcode types that are used very rarely that are not governed > by the CmdSN-based flow control - immediate commands, > SNACK, unsolicited NOP-In, Reject, and Async Messages. There is no requirement that there be an explicit wire-level protocol. Merely that the ULP establishes a mechanism by which the sender knows whether it can send a given untagged message. iSCSI CmdSN flow control already provides this flow control for most iSER packets. So the only issue is establishing rules for the remaining packets. Controlling the flow of *most* packets is *not* flow control. It is somewhat akin to having a strictly balanced budget except for these three funds which are unrestricted. > > > Note that the above does not include the unsolicited Data-out > PDUs since the worst case number of these is precisely known from > CmdSN, but the worst case buffer provisioning for these would > be both unnecessary and extremely expensive in reality. > Under-provisioning of buffers is a local issue, with the caveat that doing it improperly is a fault on the Data Sink's part. There can also be faults from exhaustion of CPU power, hardware faults and plain old software errors.The server is obviously expected to keep these to a minimum. The key distinction that must be made is between granting credits, providing buffers and matching buffers. The classic simple ordered Receive Queue is the one interface that I believe everyone agrees must be supported. With it the Data Sink ULP posts a receive buffer, and thereby grants a credit and pre-assigns a buffer to the QN/MSN. The Shared Receive Queue (proposed in draft-hilland) shares both buffers and credits across a pool. Buffers are assigned to the QN/MSN on an as needed basis. The implementation has an option of filling in buffers for the gap when a high MSN is received, otherwise the buffer is allocated when a portion of it is first received. Credits are consumed when buffers are allocated. Note that Shared Receive Queues only apply to the RDMA Send queue, the RDMA Read queue is not documented, but given that a fixed limit is configured would presumably be a simple ordered Receive Queue. Shared Buffer Pools place buffers in a pool, but assign credits on a per stream basis. If an MSN exceeds the range implied by the credits it is rejected as invalid whether there is a buffer available or not. iSER seems to call for the ability to pool credits across all streams in a session. But it would not necessarily be the same set of streams that you would want to share buffers over. There could be advantages of pooling buffers between sessions, while still tracking credits on a per session basis. In any event, these are all *local* questions. The only *wire* question is whether the Data Source can know whether or not it is legal for it to send a given untagged message. Stating that for message types "x" it is legal as long as the Data Source thinks it has a reason to send "x" is NOT flow control. For each of the "exceptional" types, what is required that a rule be derived on how many of them can be outstanding, and how the sender knows when they are no longer outstanding. If, as claimed, it is a trivial matter for the Data Sink to make these calculations, then it should be easy to enumerate these rules. > The iSER design team thus believed that most storage implementations > will use buffer pools to deal with this reality (as they have always > been), and the rare "fringe" opcode types mentioned above could > easily be dealt with in the statistical provisioning scheme of things, > being > so rare and infrequent. It is totally incorrect for an Upper Layer Protocol to be designed with presumptions as to implementation of the lower layers. If you believe buffer pools are required for the correct functioning of an application using iWARP then you should be arguing for that change to iWARP. Otherwise, the Upper Layer Protocol must be defined so as to rely upon the published protocol and nothing else. iWARP requires the ULP to take responsibility for flow control of untagged messages. Period. > Despite this belief (in fact, even before we are convinced of this > approach), > we did a diligent analysis of a Send Message flow control protocol for > iSER > - the ultimate conclusion was that it's way too much overhead to run > this > protocol, it's slow-to-respond to changing I/O loads, reclaiming of > credits > is a burdensome process, requires RTT delays to announce new credits > etc. > That is based upon the assumption that iSER flow control requires iSER flow control messages. This is not a requirement. A requirement that the Data Source MUST NOT submit more than one connection termination notice upon any given connection would fully flow control that type of message -- with no wire protocol messages being exchanged. > I believe the approach adopted in the current iSER draft is > appropriate, > we do however need to polish the flow control discussion to include > some of the design rationale. Rationale are not constraints upon the sender of untagged messages. Flow *control*, by definition, is a constraint on the sender. The constraint does not have to take the form of dynamically exchanged messages, or even per-session negotiated limits. But it does require that a limit be unambiguously identified. Otherwise it is not flow *control*. Again, this has nothing to do with how many buffers the Data Sink must provision and when. Dynamic binding of buffers is a totally valid strategy, especially if the Data Sink has "low water mark" warnings and processes responsible for responding to those alarms to restock the buffer pool. The point is that failure to provide true flow control *requires* that *all* implementations build such an infrastructure. It is taking a feature that is desirable for high volume servers and making it a de facto requirement for *all* servers. Even those who only intend to support a single client. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 30 15:38:58 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA15109 for ; Wed, 30 Jul 2003 15:38:58 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hwmO-00011t-Ru for rddp-archive@odin.ietf.org; Wed, 30 Jul 2003 15:38:32 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6UJcWQt003953 for rddp-archive@odin.ietf.org; Wed, 30 Jul 2003 15:38:32 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hwmO-00011g-Om for rddp-web-archive@optimus.ietf.org; Wed, 30 Jul 2003 15:38:32 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA15095 for ; Wed, 30 Jul 2003 15:38:28 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hwlt-0000xU-Lk; Wed, 30 Jul 2003 15:38:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hwlo-0000x9-Nd for rddp@optimus.ietf.org; Wed, 30 Jul 2003 15:37:56 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA15079 for ; Wed, 30 Jul 2003 15:37:52 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19hwln-00030q-00 for rddp@ietf.org; Wed, 30 Jul 2003 15:37:55 -0400 Received: from atlrel6.hp.com ([156.153.255.205]) by ietf-mx with esmtp (Exim 4.12) id 19hwlm-00030n-00 for rddp@ietf.org; Wed, 30 Jul 2003 15:37:54 -0400 Received: from rosemail.rose.hp.com (rosemail.rose.hp.com [15.96.64.26]) by atlrel6.hp.com (Postfix) with ESMTP id 40A021C013FB for ; Wed, 30 Jul 2003 15:37:49 -0400 (EDT) Received: from swathi (dhcp-roseor-bdj047.rose.hp.com [15.23.139.47]) by rosemail.rose.hp.com with SMTP (8.7.1/8.7.3 SMKit7.02) id MAA25173 for ; Wed, 30 Jul 2003 12:37:48 -0700 (PDT) Message-ID: <019901c356d2$0e371e40$18a4f40f@rose.hp.com> From: "Mallikarjun C." To: References: Subject: Re: [rddp] WG chair on buffer exhaustion Date: Wed, 30 Jul 2003 12:37:47 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit I recall several on this WG have agreed that the current DDP text that I pointed out is severe. Let's just fix that text to address the WG concerns. You had earlier argued that DDP text is fine the way it is, now you're proposing a _lot of_ text that does not fix the text the WG had a problem with. No amount of wordsmithing that does not fix the original text will resolve the issue I had brought up. Besides, I do not agree with all the text you suggest. Needless to say, I prefer the text I recommended. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Hewlett-Packard MS 5668 Roseville CA 95747 cbm@rose.hp.com ----- Original Message ----- From: "Caitlin Bestler" To: "Mallikarjun C." Cc: Sent: Tuesday, July 29, 2003 1:51 PM Subject: Re: [rddp] WG chair on buffer exhaustion > > > My suggestion is to change the text along the following lines: > > > > Section 3.2 to: > > " Note, however, that the ULP needs to address flow > > control issues. Any ULP that runs on DDP SHOULD > > consider the lack of an associated receive buffer to > > place an inbound Untagged Message of the ULP as an > > exception." > > > I would suggest the following section: > > Section TBD ULP Flow-control > > The ULP is responsible for providing flow control for > untagged messages. The Data Source MUST know that it > is authorized to send an untagged message before sending > it. The Data Sink MUST know the maximum number of untagged > messages that it has authorized for reception on a DDP > Stream or set of streams. > > The method by which the Data Source and Data Sink ULP > reach agreement on this limit is outside the scope of > this specification. Possible methods include dynamically > negotiated credits, session negotiated credits and > fixed limits imposed by the ULP specification. > > The Data Sink SHOULD provide sufficient buffers for > the untagged messages that it has authorized. However, > it is not required to do so by pre-committing a buffer > for each valid QN/MSN combination. However the local > interface MUST allow the Data Sink ULP to pre-specify > the buffer for each valid QN/MSN combination. > > Receipt of an untagged message that was not authorized > by the Data Sink ULP SHOULD result in termination of the > DDP Stream. The local interface may allow authorization > of untagged messages for a specific stream and/or for > a group of streams. > > Receipt of an untagged message that was authorized, but > for which there is no available buffer, MAY result in > termination of the DDP Stream. However, implementations > MAY treat the absence of a buffer for a valid untaged > message as a local fault. Implementation specific > procedures to recover from such a fault MAY be used, > especially if they avoid impacting the remote peer. > > > I believe this specifically takes care of one iSER concern. > The bulk of the allowed untagged messages have CmdSNs, and > are flow controlled for a *set* of streams that belong to a > single session. It would require that a limit be set for > asynchronous messages, but would allow that to be a ULP > constant rather than a dynamically negotiated value. > > It also allows for strict and immediate termination of > untagged messages that are bound to simple Receive Queues > or to the RDMA Read Engine. In the latter case, a "soft > limit" approach would seem to be very inappropriate. > > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 30 16:01:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15872 for ; Wed, 30 Jul 2003 16:01:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hx8B-0001ln-9J for rddp-archive@odin.ietf.org; Wed, 30 Jul 2003 16:01:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6UK13WS006797 for rddp-archive@odin.ietf.org; Wed, 30 Jul 2003 16:01:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hx8B-0001lY-69 for rddp-web-archive@optimus.ietf.org; Wed, 30 Jul 2003 16:01:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15851 for ; Wed, 30 Jul 2003 16:00:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19hx89-0003Ad-00 for rddp-web-archive@ietf.org; Wed, 30 Jul 2003 16:01:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19hx89-0003Aa-00 for rddp-web-archive@ietf.org; Wed, 30 Jul 2003 16:01:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hx89-0001l9-Ot; Wed, 30 Jul 2003 16:01:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hx7t-0001kX-KK for rddp@optimus.ietf.org; Wed, 30 Jul 2003 16:00:45 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15833 for ; Wed, 30 Jul 2003 16:00:40 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19hx7s-0003AW-00 for rddp@ietf.org; Wed, 30 Jul 2003 16:00:44 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19hx7r-0003AT-00 for rddp@ietf.org; Wed, 30 Jul 2003 16:00:43 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 0EE5F245ECF; Wed, 30 Jul 2003 16:00:22 -0400 (EDT) Date: Wed, 30 Jul 2003 15:00:09 -0500 Subject: Re: [rddp] WG chair on buffer exhaustion Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: To: "Mallikarjun C." From: Caitlin Bestler In-Reply-To: <019901c356d2$0e371e40$18a4f40f@rose.hp.com> Message-Id: <6C650C0E-C2C8-11D7-81E7-003065D48EE0@asomi.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Wednesday, July 30, 2003, at 02:37 PM, Mallikarjun C. wrote: > I recall several on this WG have agreed that the > current DDP text that I pointed out is severe. > Let's just fix that text to address the WG concerns. > > You had earlier argued that DDP text is fine the way > it is, now you're proposing a _lot of_ text that > does not fix the text the WG had a problem with. > > No amount of wordsmithing that does not fix the original > text will resolve the issue I had brought up. Besides, I do > not agree with all the text you suggest. > > Needless to say, I prefer the text I recommended. The text you proposed fails to distinguish between ULP authorized messages and ULP unauthorized messages. As has been discussed on the WG, this creates a severe security vulnerability. The wording I propose makes it clear that the receiving side has flexibility in handling a problem caused by lack of buffering *without* creating a responsibility to try to recover from unauthorized untagged messages. Can you be specific as to which portion you disagree with? And are those disagreements consistent with keeping iWARP a *reliable* protocol with ULP flow control? The bottom line is that when the Data Source sends an untagged message it is either authorized by the ULP or it is not. If it is authorized, the Data Sink is expected to receive it. Pretending not to receive it, or dropping the connections are strategies for fault-containment. That does not mean they are acceptable protocol behaviors. The wording you propose endorses such strategies. If it is not authorized, the Data Sink SHOULD reject it. It definitely MUST NOT place another session at risk by accepting an unauthorized message. The consensus on the WG is *solely* to allow a local implementation to have some flexibility in how to recover from fault conditions. Your proposed wording changes far exceed that. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 30 16:49:27 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA17189 for ; Wed, 30 Jul 2003 16:49:27 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hxsc-0003VE-B0 for rddp-archive@odin.ietf.org; Wed, 30 Jul 2003 16:49:02 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6UKn2D4013458 for rddp-archive@odin.ietf.org; Wed, 30 Jul 2003 16:49:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hxsc-0003Uz-7R for rddp-web-archive@optimus.ietf.org; Wed, 30 Jul 2003 16:49:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA17179 for ; Wed, 30 Jul 2003 16:48:56 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19hxsa-0003Sl-00 for rddp-web-archive@ietf.org; Wed, 30 Jul 2003 16:49:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19hxsZ-0003Si-00 for rddp-web-archive@ietf.org; Wed, 30 Jul 2003 16:48:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hxsa-0003Ud-OM; Wed, 30 Jul 2003 16:49:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hxrf-0003U8-VL for rddp@optimus.ietf.org; Wed, 30 Jul 2003 16:48:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA17150 for ; Wed, 30 Jul 2003 16:47:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19hxrd-0003SN-00 for rddp@ietf.org; Wed, 30 Jul 2003 16:48:01 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19hxrd-0003S4-00 for rddp@ietf.org; Wed, 30 Jul 2003 16:48:01 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 493F72453C8; Wed, 30 Jul 2003 16:47:37 -0400 (EDT) Date: Wed, 30 Jul 2003 15:46:49 -0500 Subject: Re: [rddp] WG chair on buffer exhaustion Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: To: "Mallikarjun C." From: Caitlin Bestler In-Reply-To: <012001c35609$599f4390$18a4f40f@rose.hp.com> Message-Id: Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Tuesday, July 29, 2003, at 02:39 PM, Mallikarjun C. wrote: > Section 3.2 to: > " Note, however, that the ULP needs to address flow control > issues. Any > ULP that runs on DDP SHOULD consider the lack of an associated > receive > buffer to place an inbound Untagged Message of the ULP as an > exception." If the local interface has allows the DDP endpoint to know the valid range of MSNs on a stream (which it will for simple Receive Queues at the minimum)\ then it definitely SHOULD NOT consider the lack of a buffer for an *invalid message* to be "an exception". It is a *violation* of the Upper Layer Protocol, and quite possibly represents an *attack*. It definitely SHOULD terminate the stream promptly. I am willing to accept Pat Thaler's argument that some applications will find it awkward to share credits with the DDP layer on a per-stream basis. That is why there is an "If" on the above wording. But the change you propose implies that the DDP layer SHOULD be attempting to accept untagged messages by whatever means possible and only terminate the stream as a last resort. That would be a mistake, and create a major security vulnerability. The DDP layer SHOULD reject messages known to be invalid, whether or not there are buffers available. I would go further to state that DDP MUST NOT accept invalid untagged messages if doing so would endanger any other ULP session. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 30 17:57:28 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA19825 for ; Wed, 30 Jul 2003 17:57:28 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hywR-00079B-Vi for rddp-archive@odin.ietf.org; Wed, 30 Jul 2003 17:57:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6ULv3BW027461 for rddp-archive@odin.ietf.org; Wed, 30 Jul 2003 17:57:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hywR-00078n-RI for rddp-web-archive@optimus.ietf.org; Wed, 30 Jul 2003 17:57:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA19814 for ; Wed, 30 Jul 2003 17:56:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19hywO-00049K-00 for rddp-web-archive@ietf.org; Wed, 30 Jul 2003 17:57:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19hywN-00049H-00 for rddp-web-archive@ietf.org; Wed, 30 Jul 2003 17:56:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hywP-00075e-LW; Wed, 30 Jul 2003 17:57:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19hyvz-00072q-QJ for rddp@optimus.ietf.org; Wed, 30 Jul 2003 17:56:35 -0400 Received: from mxic2.corp.emc.com (mxic2.corp.emc.com [128.221.31.40]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA19806 for ; Wed, 30 Jul 2003 17:56:29 -0400 (EDT) From: Black_David@emc.com Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Wed, 30 Jul 2003 17:50:50 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0C2467BE@corpmx14.us.dg.com> To: rddp@ietf.org Date: Wed, 30 Jul 2003 17:40:07 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain Subject: [rddp] WG chair on buffer exhaustion (again) Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , With WG Chair hat on, let me step into this debate: > > Section 3.2 to: > > " Note, however, that the ULP needs to address flow control issues. Any > > ULP that runs on DDP SHOULD consider the lack of an associated receive > > buffer to place an inbound Untagged Message of the ULP as an exception." > > If the local interface has allows the DDP endpoint to know the valid > range of MSNs on a stream (which it will for simple Receive Queues at > the minimum) then it definitely SHOULD NOT consider the lack of a > buffer for an *invalid message* to be "an exception". Two issues are getting conflated here, and need to be kept separate: - Mallikarjun's issue involves statistical (under-) provisioning of buffers, so that an otherwise valid untagged message shows up without a buffer available to receive it (if the implementation had done worst-case provisioning, there would have been a buffer available). I believe there is WG rough consensus that this is an exceptional condition that does not need to immediately cause the termination of the DDP stream that sent the message, but it SHOULD not occur frequently. Mallikarjun's proposed text is close enough to what's needed for this. This condition could happen even on a simple Receive Queue if statistical (under-) provisioning is being used. - Caitlin's issue involves a peer violating a fixed per-DDP-stream limit on the number of untagged messages allowed . At the moment, the exact need and nature of such limits is still an open issue that needs to be resolved in the context of the security draft. When we have it resolved there, additional text will (probably) get driven into the DDP draft. If the issue is resolved in the manner Caitlin advocates, then her proposed text is certainly a candidate for what the DDP draft will need to say, but at the moment, the issue is open, and the DDP draft should not change what it says about this issue until it is resolved in the context of the security draft. Also, I've seen some confusing references to iSER and assumptions about local interface and resource sharing model. Shared Receive Queues have *not* been approved/adopted by this WG (and can't be until we get the resource sharing issue [2nd bullet above] resolved). To the extent that iSER requires Shared Receive Queues and can't cope with any other local resource model, it's probably made an overly restrictive design assumption that it's designers would be well-advised to reconsider, keeping in mind that the Hilland verbs draft is at best an *example* of a local interface to RDDP; it should not be viewed as normative functional requirements on that local interface. With luck, that's useful clarification, but feel free to ask if anything above doesn't make sense ... Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 30 20:53:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA25401 for ; Wed, 30 Jul 2003 20:53:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19i1gm-0004oO-Mc for rddp-archive@odin.ietf.org; Wed, 30 Jul 2003 20:53:05 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6V0r4np018495 for rddp-archive@odin.ietf.org; Wed, 30 Jul 2003 20:53:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19i1gm-0004oE-Cb for rddp-web-archive@optimus.ietf.org; Wed, 30 Jul 2003 20:53:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA25388 for ; Wed, 30 Jul 2003 20:52:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19i1gj-0005zA-00 for rddp-web-archive@ietf.org; Wed, 30 Jul 2003 20:53:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19i1gj-0005z7-00 for rddp-web-archive@ietf.org; Wed, 30 Jul 2003 20:53:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19i1gj-0004na-BD; Wed, 30 Jul 2003 20:53:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19i1gO-0004kV-Jw for rddp@optimus.ietf.org; Wed, 30 Jul 2003 20:52:40 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA25375 for ; Wed, 30 Jul 2003 20:52:35 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19i1gL-0005yo-00 for rddp@ietf.org; Wed, 30 Jul 2003 20:52:37 -0400 Received: from atlrel6.hp.com ([156.153.255.205]) by ietf-mx with esmtp (Exim 4.12) id 19i1gL-0005yk-00 for rddp@ietf.org; Wed, 30 Jul 2003 20:52:37 -0400 Received: from rosemail.rose.hp.com (rosemail.rose.hp.com [15.96.64.26]) by atlrel6.hp.com (Postfix) with ESMTP id 4E5391C014C4; Wed, 30 Jul 2003 20:52:34 -0400 (EDT) Received: from swathi (dhcp-roseor-bdj047.rose.hp.com [15.23.139.47]) by rosemail.rose.hp.com with SMTP (8.7.1/8.7.3 SMKit7.02) id RAA26201; Wed, 30 Jul 2003 17:52:33 -0700 (PDT) Message-ID: <01bb01c356fe$06373870$18a4f40f@rose.hp.com> From: "Mallikarjun C." To: , References: Date: Wed, 30 Jul 2003 17:49:44 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Content-Transfer-Encoding: 7bit Subject: [rddp] Re: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Responding to Caitlin's two messages..... First off, thanks for the review. At a high level, I view Caitlin's comments as stressing two points - a) The iSCSI/iSER peers must each know the precise number of control-type messages to expect at any given instant. b) Either a ULP has *positive* (absolutely deterministic) Send Message flow control or it doesn't have any. I think several iSCSI practitioners (myself included) disagree with (a) - iSCSI is not designed to allow each iSCSI endnode to precisely know the number of inbound PDUs it can expect (for good reasons). Now, iSER can make an attempt to "fix" this (what the authors considered a) non-issue (for < 1% of the PDUs) with expensive protocol. The iSER designers chose not to do this. On (b), I find it very hard to agree with this "positive flow control, else you fail the test" argument. I don't know if it might make sense for IPC apps, but it doesn't make any sense for storage in my mind - particularly when the returns are negative in making it "positive". The iSER-assisted iSCSI *does* provide ULP-level flow control for 99% - in volume - of the Untagged messages (control-type PDUs). But yes, it does not currently give you a "positive" flow control. Now on to specifics - >There is an identical flow control issue for > RDMA Reads. Not true. There's a built-in credit renewal in an RDMA Read. The Peer issuing the RDMA Read knows it can reuse the Read credit when it receives an RDMA Read Response. Send Messages carrying the "fringe" iSCSI PDUs need both new wire protocol and cross-layer chit-chat within an end-node between iSCSI and iSER - in order to renew credits. >Wouldn't the appropriate rule for SNACK messages be "don't"? As you point out, SNACK isn't useful in an iSCSI/iSER context. That's why section 9.3.11 says that SNACKs "SHOULD NOT" be issued. We felt that a "MUST NOT" is too severe since SNACK is an iSCSI-to-iSCSI PDU that would be properly dealt with at the iSCSI level even when used on an iSCSI/iSER connection. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Hewlett-Packard MS 5668 Roseville CA 95747 cbm@rose.hp.com ----- Original Message ----- From: "Caitlin Bestler" To: "Mallikarjun C." Cc: ; Sent: Wednesday, July 30, 2003 9:07 AM Subject: Re: iSCSI/iWARP drafts and flow control > > On Tuesday, July 29, 2003, at 08:58 PM, Mallikarjun C. wrote: > > > As Mike points out, the CmdSN-based flow control > > in iSCSI is relevant here. Let me note that the design > > team behind the current iSER draft considered this topic > > in great detail, but I can now clearly see that the draft > > unfortunately does not capture the design rationale very well. > > > > It could be clearer. But even if it were clearer, it would > not change the fact that it fails to provide ULP-level > flow control for untagged messages. > > The requirement here is that the ULP provide *flow control* > for untagged messages. Control means that the Data Source > either has permission to send an untagged message, or it > does not. There is an identical flow control issue for > RDMA Reads. You either are allowed to send one or you > are not. > > If you are allowed to send an untagged message, you have > an expectation that the other side has the resources to > handle it. Bugs, under-provisioning and hardware faults > are all facts of life. So robust applications are prepared > to deal with faults. But faults reflect a *failure*. > > If an untagged message is allowed, then the Data Source > has ever reason to expect that the Data Sink will handle > it properly. Failure to do so if a fault on the Data Sink. > > If an untagged message is not allowed, then the Data Source > had no right to send it. It cannot complain if the Data > Sink terminates the stream. In this case the Data Source > is the one committing the fault. > > > > iSCSI does not provide a PDU-level positive flow control > > but instead relies on the CmdSN feature, from which most > > of the iSCSI (what DA/iSER call as the) "control-type" PDU traffic > > can be precisely estimated (note that only control-type PDUs > > are candidates for Send Messages and thus relevant to this > > discussion). However, it turns out that there are certain > > opcode types that are used very rarely that are not governed > > by the CmdSN-based flow control - immediate commands, > > SNACK, unsolicited NOP-In, Reject, and Async Messages. > > There is no requirement that there be an explicit wire-level > protocol. Merely that the ULP establishes a mechanism by > which the sender knows whether it can send a given untagged > message. > > iSCSI CmdSN flow control already provides this flow control > for most iSER packets. So the only issue is establishing > rules for the remaining packets. > > Controlling the flow of *most* packets is *not* flow control. > It is somewhat akin to having a strictly balanced budget > except for these three funds which are unrestricted. > > > > > > > > Note that the above does not include the unsolicited Data-out > > PDUs since the worst case number of these is precisely known from > > CmdSN, but the worst case buffer provisioning for these would > > be both unnecessary and extremely expensive in reality. > > > > Under-provisioning of buffers is a local issue, with the caveat > that doing it improperly is a fault on the Data Sink's part. There > can also be faults from exhaustion of CPU power, hardware faults > and plain old software errors.The server is obviously expected to > keep these to a minimum. > > The key distinction that must be made is between granting credits, > providing buffers and matching buffers. > > The classic simple ordered Receive Queue is the one interface that > I believe everyone agrees must be supported. With it the Data Sink > ULP posts a receive buffer, and thereby grants a credit and pre-assigns > a buffer to the QN/MSN. > > The Shared Receive Queue (proposed in draft-hilland) shares both > buffers and credits across a pool. Buffers are assigned to the > QN/MSN on an as needed basis. The implementation has an option of > filling in buffers for the gap when a high MSN is received, otherwise > the buffer is allocated when a portion of it is first received. > Credits are consumed when buffers are allocated. > > Note that Shared Receive Queues only apply to the RDMA Send queue, > the RDMA Read queue is not documented, but given that a fixed limit > is configured would presumably be a simple ordered Receive Queue. > > Shared Buffer Pools place buffers in a pool, but assign credits on > a per stream basis. If an MSN exceeds the range implied by the credits > it is rejected as invalid whether there is a buffer available or not. > > iSER seems to call for the ability to pool credits across all streams > in a session. But it would not necessarily be the same set of streams > that you would want to share buffers over. There could be advantages > of pooling buffers between sessions, while still tracking credits on > a per session basis. > > In any event, these are all *local* questions. The only *wire* question > is whether the Data Source can know whether or not it is legal for it > to send a given untagged message. > > Stating that for message types "x" it is legal as long as the Data > Source thinks it has a reason to send "x" is NOT flow control. > > For each of the "exceptional" types, what is required that a rule be > derived on how many of them can be outstanding, and how the sender > knows when they are no longer outstanding. > > If, as claimed, it is a trivial matter for the Data Sink to make > these calculations, then it should be easy to enumerate these rules. > > > The iSER design team thus believed that most storage implementations > > will use buffer pools to deal with this reality (as they have always > > been), and the rare "fringe" opcode types mentioned above could > > easily be dealt with in the statistical provisioning scheme of things, > > being > > so rare and infrequent. > > It is totally incorrect for an Upper Layer Protocol to be designed with > presumptions as to implementation of the lower layers. If you believe > buffer pools are required for the correct functioning of an application > using iWARP then you should be arguing for that change to iWARP. > > Otherwise, the Upper Layer Protocol must be defined so as to rely upon > the published protocol and nothing else. > > iWARP requires the ULP to take responsibility for flow control of > untagged messages. Period. > > > > Despite this belief (in fact, even before we are convinced of this > > approach), > > we did a diligent analysis of a Send Message flow control protocol for > > iSER > > - the ultimate conclusion was that it's way too much overhead to run > > this > > protocol, it's slow-to-respond to changing I/O loads, reclaiming of > > credits > > is a burdensome process, requires RTT delays to announce new credits > > etc. > > > > That is based upon the assumption that iSER flow control requires > iSER flow control messages. This is not a requirement. A requirement > that the Data Source MUST NOT submit more than one connection > termination > notice upon any given connection would fully flow control that type of > message -- with no wire protocol messages being exchanged. > > > > I believe the approach adopted in the current iSER draft is > > appropriate, > > we do however need to polish the flow control discussion to include > > some of the design rationale. > > Rationale are not constraints upon the sender of untagged > messages. Flow *control*, by definition, is a constraint on the > sender. The constraint does not have to take the form of dynamically > exchanged messages, or even per-session negotiated limits. But it > does require that a limit be unambiguously identified. > > Otherwise it is not flow *control*. > > Again, this has nothing to do with how many buffers the Data Sink > must provision and when. Dynamic binding of buffers is a totally > valid strategy, especially if the Data Sink has "low water mark" > warnings and processes responsible for responding to those alarms > to restock the buffer pool. > > The point is that failure to provide true flow control *requires* > that *all* implementations build such an infrastructure. It is > taking a feature that is desirable for high volume servers and > making it a de facto requirement for *all* servers. Even those > who only intend to support a single client. > > > > > Caitlin Bestler - cait@asomi.com - http://asomi.com/ > > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Jul 30 21:11:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA25657 for ; Wed, 30 Jul 2003 21:11:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19i1yB-0005Lq-US for rddp-archive@odin.ietf.org; Wed, 30 Jul 2003 21:11:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6V1B3BV020564 for rddp-archive@odin.ietf.org; Wed, 30 Jul 2003 21:11:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19i1yB-0005Lb-PJ for rddp-web-archive@optimus.ietf.org; Wed, 30 Jul 2003 21:11:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA25639 for ; Wed, 30 Jul 2003 21:10:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19i1y9-000641-00 for rddp-web-archive@ietf.org; Wed, 30 Jul 2003 21:11:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19i1y8-00063y-00 for rddp-web-archive@ietf.org; Wed, 30 Jul 2003 21:11:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19i1yA-0005L5-08; Wed, 30 Jul 2003 21:11:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19i1xv-0005Kr-UY for rddp@optimus.ietf.org; Wed, 30 Jul 2003 21:10:47 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA25631 for ; Wed, 30 Jul 2003 21:10:42 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19i1xt-00063j-00 for rddp@ietf.org; Wed, 30 Jul 2003 21:10:45 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19i1xs-00063f-00 for rddp@ietf.org; Wed, 30 Jul 2003 21:10:44 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id D9AFF24572F; Wed, 30 Jul 2003 21:10:20 -0400 (EDT) Date: Wed, 30 Jul 2003 20:10:07 -0500 Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: , To: "Mallikarjun C." From: Caitlin Bestler In-Reply-To: <01bb01c356fe$06373870$18a4f40f@rose.hp.com> Message-Id: Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Subject: [rddp] Re: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Wednesday, July 30, 2003, at 07:49 PM, Mallikarjun C. wrote: > >> There is an identical flow control issue for >> RDMA Reads. > > Not true. There's a built-in credit renewal in an RDMA Read. > The Peer issuing the RDMA Read knows it can reuse the > Read credit when it receives an RDMA Read Response. > Send Messages carrying the "fringe" iSCSI PDUs need both > new wire protocol and cross-layer chit-chat within an end-node > between iSCSI and iSER - in order to renew credits. No new wire protocol is required. Restoration of no-CmdSNs credits could be piggy-backed on top of the existing CmdSN credit system. Basically each Cmd reply restores one "Cmd Credit". It could also restore an implicit number of "NoCmd" credits. Basically if Cmd X had been proceeded by Y "NoCmd" messages, then the reply to Cmd X would restore Y "NoCmd" credits. It is simple credit counting, suitable for implementation in hardware or software at any number of layers, and requires no additional wire messages or even fields. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 31 11:47:10 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA02249 for ; Thu, 31 Jul 2003 11:47:10 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iFdc-0007n2-8f for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 11:46:44 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6VFki7A029944 for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 11:46:44 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iFdc-0007mt-4i for rddp-web-archive@optimus.ietf.org; Thu, 31 Jul 2003 11:46:44 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA02242 for ; Thu, 31 Jul 2003 11:46:39 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iFdb-0005Ez-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 11:46:43 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19iFda-0005Ew-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 11:46:42 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iFcv-0007mL-Br; Thu, 31 Jul 2003 11:46:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iFcC-0007li-VF for rddp@optimus.ietf.org; Thu, 31 Jul 2003 11:45:16 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA02231 for ; Thu, 31 Jul 2003 11:45:12 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iFcB-0005Em-00 for rddp@ietf.org; Thu, 31 Jul 2003 11:45:15 -0400 Received: from mx01.netapp.com ([198.95.226.53]) by ietf-mx with esmtp (Exim 4.12) id 19iFcB-0005EU-00 for rddp@ietf.org; Thu, 31 Jul 2003 11:45:15 -0400 Received: from frejya.corp.netapp.com (frejya [10.10.20.91]) by mx01.netapp.com (8.12.9/8.12.9/NTAP-1.5) with ESMTP id h6VFihQw023266 for ; Thu, 31 Jul 2003 08:44:43 -0700 (PDT) Received: from black.eng.netapp.com (black.eng.netapp.com [10.56.10.24]) by frejya.corp.netapp.com (8.12.9/8.12.9/NTAP-1.5) with ESMTP id h6VFigRe019864 for ; Thu, 31 Jul 2003 08:44:43 -0700 (PDT) Received: by black.eng.netapp.com with Internet Mail Service (5.5.2653.19) id ; Thu, 31 Jul 2003 08:44:41 -0700 Message-ID: From: "Kanevsky, Arkady" To: "'Black_David@emc.com'" , rddp@ietf.org Subject: RE: [rddp] WG chair on buffer exhaustion (again) Date: Thu, 31 Jul 2003 08:37:57 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , David, I am trying to comprehand the 2 issues. You stated that Mallikarjun's issue involves statistical (under-) provisioning of buffers can happen without any Shared Receive queues. 1. What is the definition of reliable connection based on this change? 2. How does UPL knows/control what is the connection behavior: connection broken(termination DDP stream) if no recv buffer is availble vs. Send message dropped on the floor and not delivered, but DDP connection stays? 3. How is the Stag invalidation effected (if at all) with both proposals? Thanks, Arkady Arkady Kanevsky email: arkady@netapp.com Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 3rd Floor http: www.netapp.com Waltham, MA 02451-2010 general phone: 781-768-5300 > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Wednesday, July 30, 2003 5:40 PM > To: rddp@ietf.org > Subject: [rddp] WG chair on buffer exhaustion (again) > > > With WG Chair hat on, let me step into this debate: > > > > Section 3.2 to: > > > " Note, however, that the ULP needs to address flow control > > > issues. > Any > > > ULP that runs on DDP SHOULD consider the lack of an > associated > receive > > > buffer to place an inbound Untagged Message of the ULP as an > exception." > > > > If the local interface has allows the DDP endpoint to know > the valid > > range of MSNs on a stream (which it will for simple Receive > Queues at > > the minimum) then it definitely SHOULD NOT consider the lack of a > > buffer for an *invalid message* to be "an exception". > > Two issues are getting conflated here, and need to be kept separate: > - Mallikarjun's issue involves statistical (under-) > provisioning of buffers, > so that an otherwise valid untagged message shows up > without a buffer > available to receive it (if the implementation had done > worst-case > provisioning, there would have been a buffer > available). I believe > there is WG rough consensus that this is an exceptional > condition > that does not need to immediately cause the termination > of the DDP > stream that sent the message, but it SHOULD not occur > frequently. > Mallikarjun's proposed text is close enough to what's > needed for this. > This condition could happen even on a simple Receive Queue if > statistical (under-) provisioning is being used. > - Caitlin's issue involves a peer violating a fixed > per-DDP-stream limit on > the number of untagged messages allowed . At the > moment, the exact > need and nature of such limits is still an open issue > that needs to be > resolved in the context of the security draft. When we > have it resolved > there, additional text will (probably) get driven into > the DDP draft. > If the issue is resolved in the manner Caitlin advocates, then > her proposed text is certainly a candidate for what the > DDP draft > will need to say, but at the moment, the issue is open, and the > DDP draft should not change what it says about this issue until > it is resolved in the context of the security draft. > > Also, I've seen some confusing references to iSER and > assumptions about local interface and resource sharing model. > Shared Receive Queues have > *not* been approved/adopted by this WG (and can't be until we > get the resource sharing issue [2nd bullet above] resolved). > To the extent that iSER requires Shared Receive Queues and > can't cope with any other local resource model, it's probably > made an overly restrictive design assumption that it's > designers would be well-advised to reconsider, keeping in > mind that the Hilland verbs draft is at best an *example* of > a local interface to RDDP; it should not be viewed as > normative functional requirements on that local interface. > > With luck, that's useful clarification, but feel free to ask > if anything above doesn't make sense ... > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > black_david@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 31 12:53:27 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA04055 for ; Thu, 31 Jul 2003 12:53:27 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iGfm-0002XB-K4 for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 12:53:02 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6VGr2iO009741 for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 12:53:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iGfm-0002X2-HG for rddp-web-archive@optimus.ietf.org; Thu, 31 Jul 2003 12:53:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA04010 for ; Thu, 31 Jul 2003 12:52:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iGfk-0005fT-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 12:53:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19iGfk-0005fP-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 12:53:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iGfk-0002Wc-Cz; Thu, 31 Jul 2003 12:53:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iGfC-0002NH-Jk for rddp@optimus.ietf.org; Thu, 31 Jul 2003 12:52:26 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA03965 for ; Thu, 31 Jul 2003 12:52:21 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iGfA-0005ey-00 for rddp@ietf.org; Thu, 31 Jul 2003 12:52:24 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19iGf5-0005en-00 for rddp@ietf.org; Thu, 31 Jul 2003 12:52:24 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id B7DC42460B1; Thu, 31 Jul 2003 12:51:27 -0400 (EDT) Date: Thu, 31 Jul 2003 11:51:14 -0500 Subject: Re: [rddp] WG chair on buffer exhaustion (again) Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: rddp@ietf.org To: "Kanevsky, Arkady" From: Caitlin Bestler In-Reply-To: Message-Id: <32872945-C377-11D7-A9DC-003065D48EE0@asomi.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Thursday, July 31, 2003, at 10:37 AM, Kanevsky, Arkady wrote: > > 2. How does ULP knows/control what is the connection behavior: > connection broken(termination DDP stream) if no recv buffer is > available vs. Send message dropped on the floor and not delivered, > but DDP connection stays? The drop, if done, must be done transparently to the ULP. That means it is being "dropped" temporarily, requesting that the LLP do something to delay delivery. Dropping the packet at the LLP layer is one option that the implementation might consider, but it is the *last* one that it should use before simply terminating the connection/stream. The primary difference of opinion is not on whether such efforts are commendable or not, but when they should be applied. I believe that they MUST NOT / SHOULD NOT be applied for invalid untagged messages, i.e. those that would have been dropped by a conventional non-sharing Receive Queue. The contrary opinion seems to favor capturing as many buffers as possible, and leaving enforcement entirely to the ULP. > 3. How is the Stag invalidation effected (if at all) with both > proposals? STag invalidation occurs at the RDMAP layer, *after* delivery from DDP. DDP delivery follows placement. So STag invalidation cannot occur unless there is a buffer available. If the lack of a buffer is not corrected, somehow, then the connection/stream will be terminated. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 31 13:31:33 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05264 for ; Thu, 31 Jul 2003 13:31:33 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iHGb-0004Ln-T6 for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 13:31:06 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6VHV5Dm016717 for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 13:31:05 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iHGb-0004LX-Pl for rddp-web-archive@optimus.ietf.org; Thu, 31 Jul 2003 13:31:05 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05244 for ; Thu, 31 Jul 2003 13:31:01 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iHGZ-0005uQ-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 13:31:03 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19iHGZ-0005uN-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 13:31:03 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iHGY-0004Jv-8j; Thu, 31 Jul 2003 13:31:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iHGG-0004Iz-9c for rddp@optimus.ietf.org; Thu, 31 Jul 2003 13:30:44 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA05237 for ; Thu, 31 Jul 2003 13:30:40 -0400 (EDT) From: pat_thaler@agilent.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iHGE-0005uE-00 for rddp@ietf.org; Thu, 31 Jul 2003 13:30:42 -0400 Received: from msgbas1x.cos.agilent.com ([192.25.240.36]) by ietf-mx with esmtp (Exim 4.12) id 19iHGD-0005uB-00 for rddp@ietf.org; Thu, 31 Jul 2003 13:30:41 -0400 Received: from relcos2.cos.agilent.com (relcos2.cos.agilent.com [130.29.152.237]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id B30931B756; Thu, 31 Jul 2003 11:30:38 -0600 (MDT) Received: from axcsbh4.cos.agilent.com (axcsbh4.cos.agilent.com [130.29.152.145]) by relcos2.cos.agilent.com (Postfix) with SMTP id EA5CA30; Thu, 31 Jul 2003 11:30:40 -0600 (MDT) Received: from 130.29.152.145 by axcsbh4.cos.agilent.com (InterScan E-Mail VirusWall NT); Thu, 31 Jul 2003 11:30:36 -0600 Received: by axcsbh4.cos.agilent.com with Internet Mail Service (5.5.2653.19) id <3NVK5GRZ>; Thu, 31 Jul 2003 11:30:35 -0600 Message-ID: <1BEBA5E8600DD4119A50009027AF54A0132E0460@axcs04.cos.agilent.com> To: cait@asomi.com, cbm@rose.hp.com Cc: ips@ece.cmu.edu, rddp@ietf.org Date: Thu, 31 Jul 2003 11:30:33 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] RE: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Caitlin, There are multiple problems with the mechanism you suggest. For one thing, the credit for additional command window in iSCSI isn't based on command replys. There is an explicit field for MaxCmdSN. Having received and finished processing a command doesn't mean that one has made space for receiving another command. The target might choose to use that space for another session or for something else altogether. Secondly, having finished a processing a command doesn't mean that other activities have completed. If we did what you suggest there might be times when a response had to be delayed because the command had been finished but the credits for the non-command messages were not yet ready. This could adversely effect performance. Also, commands can involve lengthy actions such as large data transfers. Therefore, there might be times when one is not ready to send a command reply and immediate commands or other non-command related traffic are delayed because of lack of credit. Third, it is an accounting nightmare for the implementations. Instead of just tracking a simple credit counter and updating it, they have to keep track of how many credits belong to each command. Forth, the suggestion only takes care of one direction. MaxCmdSN goes from target to initiator, but there are types of PDUs not covered by that mechanisms that go each direction. Fifth, it may have have a deadlock. If the command window has closed, the only way the target can send more credit is to send a NOP PDU. If the target doesn't have a credit, it won't be able to send the NOP PDU. It isn't a good idea to complicate implementations excessively to avoid additions to wire protocol. If one adds a flow control mechanism, it should have provision to avoid deadlock. Pat -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Wednesday, July 30, 2003 6:10 PM To: Mallikarjun C. Cc: ips@ece.cmu.edu; rddp@ietf.org Subject: Re: iSCSI/iWARP drafts and flow control On Wednesday, July 30, 2003, at 07:49 PM, Mallikarjun C. wrote: > >> There is an identical flow control issue for >> RDMA Reads. > > Not true. There's a built-in credit renewal in an RDMA Read. > The Peer issuing the RDMA Read knows it can reuse the > Read credit when it receives an RDMA Read Response. > Send Messages carrying the "fringe" iSCSI PDUs need both > new wire protocol and cross-layer chit-chat within an end-node > between iSCSI and iSER - in order to renew credits. No new wire protocol is required. Restoration of no-CmdSNs credits could be piggy-backed on top of the existing CmdSN credit system. Basically each Cmd reply restores one "Cmd Credit". It could also restore an implicit number of "NoCmd" credits. Basically if Cmd X had been proceeded by Y "NoCmd" messages, then the reply to Cmd X would restore Y "NoCmd" credits. It is simple credit counting, suitable for implementation in hardware or software at any number of layers, and requires no additional wire messages or even fields. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 31 14:00:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA06331 for ; Thu, 31 Jul 2003 14:00:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iHid-0005id-7G for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 14:00:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6VI03FY021977 for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 14:00:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iHid-0005iO-3B for rddp-web-archive@optimus.ietf.org; Thu, 31 Jul 2003 14:00:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA06304 for ; Thu, 31 Jul 2003 13:59:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iHia-0006AL-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 14:00:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19iHia-0006AI-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 14:00:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iHib-0005i7-7A; Thu, 31 Jul 2003 14:00:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iHht-0005hR-4d for rddp@optimus.ietf.org; Thu, 31 Jul 2003 13:59:17 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA06276 for ; Thu, 31 Jul 2003 13:59:12 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iHhq-0006A5-00 for rddp@ietf.org; Thu, 31 Jul 2003 13:59:14 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19iHhq-00069w-00 for rddp@ietf.org; Thu, 31 Jul 2003 13:59:14 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 4C015246036; Thu, 31 Jul 2003 13:58:54 -0400 (EDT) Date: Thu, 31 Jul 2003 12:58:40 -0500 Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: cbm@rose.hp.com, ips@ece.cmu.edu, rddp@ietf.org To: pat_thaler@agilent.com From: Caitlin Bestler In-Reply-To: <1BEBA5E8600DD4119A50009027AF54A0132E0460@axcs04.cos.agilent.com> Message-Id: <9E445BB4-C380-11D7-A9DC-003065D48EE0@asomi.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Subject: [rddp] Re: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Thursday, July 31, 2003, at 12:30 PM, pat_thaler@agilent.com wrote: > Caitlin, > > There are multiple problems with the mechanism you suggest. > > For one thing, the credit for additional command window in iSCSI isn't > based on command replys. There is an explicit field for MaxCmdSN. > Having received and finished processing a command doesn't mean that > one has made space for receiving another command. The target might > choose to use that space for another session or for something else > altogether. > There is a restocking mechanism in place. I oversimplified in my description of it. The point is that the ULP already knows how many CmdSNs that it is allowed to send. It knows when that count is depleted, and when it is restored. > Secondly, having finished a processing a command doesn't mean that > other activities have completed. If we did what you suggest there > might be times when a response had to be delayed because the command > had been finished but the credits for the non-command messages were > not yet ready. This could adversely effect performance. Also, commands > can involve lengthy actions such as large data transfers. Therefore, > there might be times when one is not ready to send a command reply and > immediate commands or other non-command related traffic are delayed > because of lack of credit. > There is no requirement that the asynch processing have completed, just that the buffers have been restocked. If the number of asynch commands is light, as claimed, crediting enough to cover the delay of a long command should not be a problem. The goal is to grant credits, nobody is requiring buffers be pre-committed for the entire lifespan of the credit. > Third, it is an accounting nightmare for the implementations. Instead > of just tracking a simple credit counter and updating it, they have to > keep track of how many credits belong to each command. > A nightmare? At worst it is a second count. One implementation could be as simple as setting a flag that will be cleared after N cmds. > Fourth, the suggestion only takes care of one direction. MaxCmdSN goes > from target to initiator, but there are types of PDUs not covered by > that mechanisms that go each direction. > Then how are they being flow controlled? > Fifth, it may have have a deadlock. If the command window has closed, > the only way the target can send more credit is to send a NOP PDU. If > the target doesn't have a credit, it won't be able to send the NOP > PDU. How is this prevented with iSCSI over TCP? What iWARP is requiring is that the ULP provide flow control for untagged buffers, to replace the flow control that TCP/SCTP provided for transport buffers. This is a *benefit*, it prevents head-of-line blocking of tagged transfers. But in any situation, if the ULP could have a deadlock situation where there are no untagged buffers available at either end then TCP/SCTP could have had a deadlock on transport buffering. > > It isn't a good idea to complicate implementations excessively to > avoid additions to wire protocol. If one adds a flow control > mechanism, it should have provision to avoid deadlock. > > Pat Meanwhile you have not addressed any of the critical problems with the lack of ULP flow control. Without flow control the sender must *guess* at how many messages can be sent. If they guess wrong the stream is terminated. Stream termination is NOT "flow control", it is either a *sanction* for failing to conform to the ULP, or it is the result of a *fault* on the receiving side. There is, by definition, no reliable mechanism to "guess" how many messages may be sent that does not produce stream terminations. That is unacceptable in something that is supposed to be a reliable protocol. If you don't want the semantics of a reliable protocol then propose an adaptation that runs over UDP. Otherwise, you cannot have a reliable protocol without *some* form of flow control on buffer usage. You cannot require the Data Sink to provide an *indefinite* amount of buffering. The arguments you cite seem to prove that iSER cannot be implemented over a standard RNIC. I believe it can be, but that requires documenting the rules that will *regulate* transmission of *all* untagged messages. I believe this can be done in a way that meets iSCSI's requirements without violating iWARP's rules. If iSCSI works over TCP, where its messages can be flow controlled by the TCP window, then it can work over a definition of iSER that complies with iWARPs rules and the definition "reliable transport". _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 31 14:23:09 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA07063 for ; Thu, 31 Jul 2003 14:23:09 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iI4Y-0006lg-8E for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 14:22:42 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6VIMg7b026010 for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 14:22:42 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iI4Y-0006lR-3s for rddp-web-archive@optimus.ietf.org; Thu, 31 Jul 2003 14:22:42 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA07022 for ; Thu, 31 Jul 2003 14:21:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iI3r-0006Js-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 14:21:59 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19iI3r-0006Jp-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 14:21:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iI3t-0006iy-13; Thu, 31 Jul 2003 14:22:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iI3S-0006ik-Ne for rddp@optimus.ietf.org; Thu, 31 Jul 2003 14:21:34 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA07005 for ; Thu, 31 Jul 2003 14:21:30 -0400 (EDT) From: pat_thaler@agilent.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iI3Q-0006JU-00 for rddp@ietf.org; Thu, 31 Jul 2003 14:21:32 -0400 Received: from msgbas1x.cos.agilent.com ([192.25.240.36]) by ietf-mx with esmtp (Exim 4.12) id 19iI3P-0006JR-00 for rddp@ietf.org; Thu, 31 Jul 2003 14:21:31 -0400 Received: from relcos2.cos.agilent.com (relcos2.cos.agilent.com [130.29.152.237]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id C9206160C4; Thu, 31 Jul 2003 12:21:28 -0600 (MDT) Received: from axcsbh4.cos.agilent.com (axcsbh4.cos.agilent.com [130.29.152.145]) by relcos2.cos.agilent.com (Postfix) with SMTP id 0B01430; Thu, 31 Jul 2003 12:21:31 -0600 (MDT) Received: from 130.29.152.145 by axcsbh4.cos.agilent.com (InterScan E-Mail VirusWall NT); Thu, 31 Jul 2003 12:21:30 -0600 Received: by axcsbh4.cos.agilent.com with Internet Mail Service (5.5.2653.19) id <3NVK5KBA>; Thu, 31 Jul 2003 12:21:30 -0600 Message-ID: <1BEBA5E8600DD4119A50009027AF54A0132E0463@axcs04.cos.agilent.com> To: cait@asomi.com, pat_thaler@agilent.com Cc: cbm@rose.hp.com, ips@ece.cmu.edu, rddp@ietf.org Date: Thu, 31 Jul 2003 12:21:29 -0600 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] RE: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Thursday, July 31, 2003 10:59 AM To: pat_thaler@agilent.com Cc: cbm@rose.hp.com; ips@ece.cmu.edu; rddp@ietf.org Subject: Re: iSCSI/iWARP drafts and flow control On Thursday, July 31, 2003, at 12:30 PM, pat_thaler@agilent.com wrote: > Caitlin, > > There are multiple problems with the mechanism you suggest. > > For one thing, the credit for additional command window in iSCSI isn't > based on command replys. There is an explicit field for MaxCmdSN. > Having received and finished processing a command doesn't mean that > one has made space for receiving another command. The target might > choose to use that space for another session or for something else > altogether. > There is a restocking mechanism in place. I oversimplified in my description of it. The point is that the ULP already knows how many CmdSNs that it is allowed to send. It knows when that count is depleted, and when it is restored. > Secondly, having finished a processing a command doesn't mean that > other activities have completed. If we did what you suggest there > might be times when a response had to be delayed because the command > had been finished but the credits for the non-command messages were > not yet ready. This could adversely effect performance. Also, commands > can involve lengthy actions such as large data transfers. Therefore, > there might be times when one is not ready to send a command reply and > immediate commands or other non-command related traffic are delayed > because of lack of credit. > There is no requirement that the asynch processing have completed, just that the buffers have been restocked. If the number of asynch commands is light, as claimed, crediting enough to cover the delay of a long command should not be a problem. The goal is to grant credits, nobody is requiring buffers be pre-committed for the entire lifespan of the credit. > Third, it is an accounting nightmare for the implementations. Instead > of just tracking a simple credit counter and updating it, they have to > keep track of how many credits belong to each command. > A nightmare? At worst it is a second count. One implementation could be as simple as setting a flag that will be cleared after N cmds. > Fourth, the suggestion only takes care of one direction. MaxCmdSN goes > from target to initiator, but there are types of PDUs not covered by > that mechanisms that go each direction. > Then how are they being flow controlled? > Fifth, it may have have a deadlock. If the command window has closed, > the only way the target can send more credit is to send a NOP PDU. If > the target doesn't have a credit, it won't be able to send the NOP > PDU. How is this prevented with iSCSI over TCP? What iWARP is requiring is that the ULP provide flow control for untagged buffers, to replace the flow control that TCP/SCTP provided for transport buffers. This is a *benefit*, it prevents head-of-line blocking of tagged transfers. But in any situation, if the ULP could have a deadlock situation where there are no untagged buffers available at either end then TCP/SCTP could have had a deadlock on transport buffering. > > It isn't a good idea to complicate implementations excessively to > avoid additions to wire protocol. If one adds a flow control > mechanism, it should have provision to avoid deadlock. > > Pat Meanwhile you have not addressed any of the critical problems with the lack of ULP flow control. Without flow control the sender must *guess* at how many messages can be sent. If they guess wrong the stream is terminated. Stream termination is NOT "flow control", it is either a *sanction* for failing to conform to the ULP, or it is the result of a *fault* on the receiving side. There is, by definition, no reliable mechanism to "guess" how many messages may be sent that does not produce stream terminations. That is unacceptable in something that is supposed to be a reliable protocol. If you don't want the semantics of a reliable protocol then propose an adaptation that runs over UDP. Otherwise, you cannot have a reliable protocol without *some* form of flow control on buffer usage. You cannot require the Data Sink to provide an *indefinite* amount of buffering. The arguments you cite seem to prove that iSER cannot be implemented over a standard RNIC. I believe it can be, but that requires documenting the rules that will *regulate* transmission of *all* untagged messages. I believe this can be done in a way that meets iSCSI's requirements without violating iWARP's rules. If iSCSI works over TCP, where its messages can be flow controlled by the TCP window, then it can work over a definition of iSER that complies with iWARPs rules and the definition "reliable transport". _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 31 15:43:42 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA10969 for ; Thu, 31 Jul 2003 15:43:42 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iJKV-00026D-K1 for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 15:43:16 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6VJhFPS008065 for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 15:43:15 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iJKV-000260-Ga for rddp-web-archive@optimus.ietf.org; Thu, 31 Jul 2003 15:43:15 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA10966 for ; Thu, 31 Jul 2003 15:43:11 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iJKH-0006th-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 15:43:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19iJKH-0006td-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 15:43:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iJKH-00025A-8i; Thu, 31 Jul 2003 15:43:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iJJO-00024P-Bq for rddp@optimus.ietf.org; Thu, 31 Jul 2003 15:42:06 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA10903 for ; Thu, 31 Jul 2003 15:42:02 -0400 (EDT) From: pat_thaler@agilent.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iJJM-0006sy-00 for rddp@ietf.org; Thu, 31 Jul 2003 15:42:04 -0400 Received: from msgbas1x.cos.agilent.com ([192.25.240.36]) by ietf-mx with esmtp (Exim 4.12) id 19iJJM-0006sv-00 for rddp@ietf.org; Thu, 31 Jul 2003 15:42:04 -0400 Received: from relcos2.cos.agilent.com (relcos2.cos.agilent.com [130.29.152.237]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id B36B71C11D; Thu, 31 Jul 2003 13:42:01 -0600 (MDT) Received: from wcosbh22.cos.agilent.com (wcosbh22.cos.agilent.com [130.29.152.178]) by relcos2.cos.agilent.com (Postfix) with ESMTP id 026151F; Thu, 31 Jul 2003 13:42:04 -0600 (MDT) content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 Date: Thu, 31 Jul 2003 13:42:00 -0600 Message-ID: <1BEBA5E8600DD4119A50009027AF54A0132E0465@axcs04.cos.agilent.com> Thread-Topic: iSCSI/iWARP drafts and flow control Thread-Index: AcNXjWQfPYoXpsOAEdesjwCQJ6pbUAAA9FVw To: , Cc: , , Content-Transfer-Encoding: quoted-printable Subject: [rddp] RE: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable All, sorry for the empty reply - I'm not sure how that happened. Caitlin, You asked some questions about how the other messages are flow = controlled in iSCSI over TCP. The answer is that they aren't flow = controlled. If iSCSI gets a PDU it cannot handle, it drops it and there = are provisions to trigger it to be resent depending on the kind of = recovery level supported. The only control for PDUs to the target is on = non-immediate commands (both SCSI Command and Task Management Function = Requeset PDUs). Note that when unsolicited non-immediate data is = permitted, iSCSI allows the command to generate a command PDU plus an = unknown number of SCSI Data-out PDUs to carry the unsolicted data. For = iSER, we require that the unsolicted SCSI Data-out PDUs be full when = there is enough unsolicted data to fill them (and we created a key to = negotiate that size). Therefore, when operating over iSER the target = does know the maximum number of PDUs that the initiator might send per = SCSI command. There is no deadlock in existing iSCSI because there is no flow control = on NOP-In and the target can always send a NOP-In to advance MaxCmdSN. To summarize, in current iSCSI, each opening in CmdSN window allows from = 1 to ? PDUs while in iSCSI over iSER, each opening in CmdSN window = allows from 1 to n PDUs where n is the amount of unsolicited data = divided by data per PDU (rounded up of course). Note also that the CmdSN window is across a session. If you have = connections in a session that are running over separate RNICs and are = using CmdSN for flow control, each RNIC will have to have access to = enough buffers for the whole window to land on it.=20 Between these two factors, CmdSN flow control will require over = provisioning buffers much of the time. Perhaps memory is cheap enough = that for an RNIC with a small number of connections this is acceptable = in exchange for using an existing mechanism. On the other hand, we will = have to create a mechanism to handle immediate commands and other PDUs = that aren't covered by CmdSN so it isn't clear to me whether this is the = right answer. The downside is overprovisioning buffers because of = sessions spanning adapters and because each command might be a write = with unsolicited data but many commands are reads. The upside is that = CmdSN window can be managed to respond to changes in load while one has = a less responsive simple mechanism to deal with the rest of the traffic. From target to initiator, the initiator knows that each command will = generate a response PDU so it can provision that before it sends a = command.=20 Login and text negotiation PDUs both ways (other than perhaps the first = PDU to open text negotiation) also have a form of iSCSI flow control. = For these, having sent a PDU, one can't send another until one has = gotten the response. R2T PDUs are replaced by RDMA Reads in iSER so they are coverd by the = RDMA Read flow control. SCSI Data-out for solicited and SCSI Data-in = PDUs are replaced by RDMA operations so we don't have to cover them. What isn't flow controlled by iSCSI: initiator to target: immediate command PDUs - existing iSCSI allows for the target to drop = these if it gets more than it can handle and the initiator can only = count on buffering for two, but the initiator can send more than that = and hope the target has buffering. One can't count on how many of these = there might be. the first Text Request PDU - there can only be one SNACK request - in theory, one shouldn't need to send this when = operating over iSER, but an initiator might send one if a timeout = occurs. NOP-Out - one doesn't expect a lot of them, but there is no limit placed = on them.=20 target to initiator: Asynchronous Message NOP-In - one doesn't expect a lot of them, but there is no limit placed = on them. While there usually won't be a lot of these PDUs, one has no way to put = a maximum number on how many there will be. There may be times when these PDUs are being sent and there is little or = no command traffic. Also the replenishment of buffers for these has = little relationship to the command processing. So I don't think one = should link the replenishment to command related activities, e.g. = advances in MaxCmdSN or reception of a command response. =20 That is the problem space. Does one put a flow control into iSER that looks at opcode and, for = commands, at the I bit to control just the PDUs that aren't limited by = MaxCmdSN or does one put in a mechanism that covers all PDUs going = through iSER? If one does the former, what happens when the transmitting iSER gets a = PDU it have to stop because of the flow control and there are MaxCmdSN = controlled PDUs after it?=20 Is iSCSI suppose to be flow control aware and manage the PDUs so that = doesn't happen?=20 Does the whole connection transmit stop until more credit arrives? Are the MaxCmdSN controlled PDUs allowed to pass the flow controlled PDU = and be transmitted? Is there a mechanism to disable flow control when the receiver doesn't = require it, e.g. large shared buffer pool with statistical provisioning? Pat _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 31 16:26:31 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA12871 for ; Thu, 31 Jul 2003 16:26:31 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iJzx-00042K-8c for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 16:26:05 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h6VKQ5OZ015516 for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 16:26:05 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iJzx-00042B-3y for rddp-web-archive@optimus.ietf.org; Thu, 31 Jul 2003 16:26:05 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA12859 for ; Thu, 31 Jul 2003 16:25:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iJzt-0007Mn-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 16:26:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19iJzt-0007Mk-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 16:26:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iJzt-00041b-JG; Thu, 31 Jul 2003 16:26:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iJzo-00041F-6K for rddp@optimus.ietf.org; Thu, 31 Jul 2003 16:25:56 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA12852 for ; Thu, 31 Jul 2003 16:25:51 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iJzm-0007Mb-00 for rddp@ietf.org; Thu, 31 Jul 2003 16:25:54 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19iJzh-0007MT-00 for rddp@ietf.org; Thu, 31 Jul 2003 16:25:53 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 05123244B80; Thu, 31 Jul 2003 16:25:29 -0400 (EDT) Date: Thu, 31 Jul 2003 15:25:15 -0500 Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: , , To: From: Caitlin Bestler In-Reply-To: <1BEBA5E8600DD4119A50009027AF54A0132E0465@axcs04.cos.agilent.com> Message-Id: <1876B2EB-C395-11D7-B621-003065D48EE0@asomi.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Subject: [rddp] Re: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Thursday, July 31, 2003, at 02:42 PM, wrote: > All, sorry for the empty reply - I'm not sure how that happened. > > Caitlin, > > You asked some questions about how the other messages are flow > controlled in iSCSI over TCP. The answer is that they aren't flow > controlled. If iSCSI gets a PDU it cannot handle, it drops it and > there are provisions to trigger it to be resent depending on the kind > of recovery level supported. The only control for PDUs to the target > is on non-immediate commands (both SCSI Command and Task Management > Function Requeset PDUs). Note that when unsolicited non-immediate data > is permitted, iSCSI allows the command to generate a command PDU plus > an unknown number of SCSI Data-out PDUs to carry the unsolicted data. > For iSER, we require that the unsolicted SCSI Data-out PDUs be full > when there is enough unsolicted data to fill them (and we created a > key to negotiate that size). Therefore, when operating over iSER the > target does know the maximum number of PDUs that the initiator might > send per SCSI command. > > There is no deadlock in existing iSCSI because there is no flow > control on NOP-In and the target can always send a NOP-In to advance > MaxCmdSN. > > To summarize, in current iSCSI, each opening in CmdSN window allows > from 1 to ? PDUs while in iSCSI over iSER, each opening in CmdSN > window allows from 1 to n PDUs where n is the amount of unsolicited > data divided by data per PDU (rounded up of course). On the contrary, existing iSCSI has buffer flow control. It runs over TCP. The receiving TCP stack declares a buffer window which the sending TCP MUST comply with. (And it SHOULD have enough buffers to match its promises, but that's a separate issue, a TCP stack can under-provision for the same reasons that the ULP finds it valuable). Even if a TCP segment will be recognized by a rototilled receiver, and its payload placed directly into a user buffer, the sending TCP is still flow controlled by the buffer window. The TCP window advertisement is not conditional. It is "I will accept N bytes". Not "n bytes as long as 90% of them can be directly placed.". This results in head-of-line blocking. A limited supply of general purpose buffering can prevent messages from being sent that would have bypassed those buffers. In order to allow DDP to be implemented efficiently, it must be able to assume that it will be able to place data as soon as it accepts a segment/chunk from the LLP for placement. The DDP layer does not do buffering. In order for this to work, the role of SCTP/TCP buffer windows MUST be replaced by ULP flow control. SCTP/TCP buffer windows are designed to ensure that there is a place to accept each received buffer (and to slow down the sender so that this condition can be maintained). Tagged messages have a valid target, or the stream is terminated. There is no condition where a valid tagged message will lack a target buffer. Untagged messages, however consume resources. Without flow control the sender can send messages which will not have a buffer to receive them. A reliable protocol prevents this with flow control. The only change from iSCSI directly over TCP and iSER is that this flow control has been refined to avoid false head-of-line blocking. But doing that requires shifting the mechanics of the flow control from the LLP to the ULP. There is no reason for iSER flow control to stall transmission of any untagged message that would not have been stall by SCTP/TCP buffer windows. In fact, it should be able to avoid false blocking. If iSCSI really required a command to be sent *now*, it would not work over TCP. Since it does, there is obviously a solution where the iSER layer would on occassion stall an untagged message on the transmit side. Your analysis consistently focuses on the receive side. Flow control is not about the receive side, it is about limiting transmit side based upon feedback from the receive side. What has to be done is to accept that constraint, and then determine the most efficient form of feedback available. It works over TCP, which is fairly crude in terms of feedback. Therefore a solution is possible. If you do not want to rely upon implicit buffer freeing, a simple flag could request an explicit ack. It would only be required under special circumstances. If it were required more often then the whole idea that this could have been estimated on the receiving side would be suspect. So far, I haven't questioned that receiver estimation would not work most of the time -- just that doing so is not flow control. It is not a reliable protocol, which means that in the *long run* it will not be robust. Unreliable protocols can be made to work quite well, with amazingly few drops and high performance -- until somebody changes one end radically and/or the network topology. Reliable protocols are supposed to prevent that. > Note also that the CmdSN window is across a session. > If you have connections in a session that are running > over separate RNICs and are using CmdSN for flow control, > each RNIC will have to have access to enough buffers > for the whole window to land on it. > This is a valid reason why the credits cannot always be enforced by the DDP layer. I have already agreed that the DDP layer cannot enforce credit limits if it does not know them, and that there are specialized cases where the ULP would not find it desirable/convenient to share this information. But the *existence* of a limit is independent of whether the receiving DDP is involved in its enforcement. The critical factor is that the Data Source ULP is aware of the limit. > Between these two factors, CmdSN flow control will require over > provisioning buffers much of the time. Perhaps memory is cheap > enough that for an RNIC with a small number of connections this > is acceptable in exchange for using an existing mechanism. On the > other hand, we will have to create a mechanism to handle immediate > commands and other PDUs that aren't covered by CmdSN so it isn't > clear to me whether this is the right answer. The downside is > overprovisioning buffers because of sessions spanning adapters and > because each command might be a write with unsolicited data but many > commands are reads. The upside is that CmdSN window can be managed > to respond to changes in load while one has a less responsive simple > mechanism to deal with the rest of the traffic. > Just as with TCP/SCTP, actual provisioning of buffers is independent of the advertised flow control. With the caveat that the advertised flow control is expected to be reasonably reliable. But buffers can be under-provisioned with amazing accurately at any protocol layer. > What isn't flow controlled by iSCSI: > initiator to target: > immediate command PDUs - existing iSCSI allows for the target to > drop these if it gets more than it can handle and the initiator > can only count on buffering for two, but the initiator can send > more than that and hope the target has buffering. One can't count > on how many of these there might be. An iSCSI can drop these under a properly flow controlled iSER as well. But it has to receive the requests first. Are they being delivered over a reliable protocol or not? Deciding to "drop" a command at the ULP layer just means that the buffer is returned to the pool quickly. It does not mean that there didn't need to be a buffer to receive the command. > > Is there a mechanism to disable flow control when the > receiver doesn't require it, e.g. large shared buffer > pool with statistical provisioning? > That would be an argument for allowing a session to explicitly negotiate these "extraneous" credits. If you have a large shared buffer, simply grant more credits. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 31 20:45:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20169 for ; Thu, 31 Jul 2003 20:45:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iO2Y-0004jZ-Li for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 20:45:02 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h710j2Zg018191 for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 20:45:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iO2Y-0004jK-IU for rddp-web-archive@optimus.ietf.org; Thu, 31 Jul 2003 20:45:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20162 for ; Thu, 31 Jul 2003 20:44:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iO2W-00015k-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 20:45:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19iO2V-00015h-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 20:44:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iO2W-0004iq-K7; Thu, 31 Jul 2003 20:45:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iO25-0004iT-GT for rddp@optimus.ietf.org; Thu, 31 Jul 2003 20:44:33 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20154 for ; Thu, 31 Jul 2003 20:44:29 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iO23-00015b-00 for rddp@ietf.org; Thu, 31 Jul 2003 20:44:31 -0400 Received: from palrel12.hp.com ([156.153.255.237]) by ietf-mx with esmtp (Exim 4.12) id 19iO22-00015Y-00 for rddp@ietf.org; Thu, 31 Jul 2003 20:44:30 -0400 Received: from rosemail.rose.hp.com (rosemail.rose.hp.com [15.96.64.26]) by palrel12.hp.com (Postfix) with ESMTP id B34981C01D4A; Thu, 31 Jul 2003 17:44:23 -0700 (PDT) Received: from swathi (dhcp-roseor-bdj047.rose.hp.com [15.23.139.47]) by rosemail.rose.hp.com with SMTP (8.7.1/8.7.3 SMKit7.02) id RAA02710; Thu, 31 Jul 2003 17:44:18 -0700 (PDT) Message-ID: <020c01c357c6$0a21e340$18a4f40f@rose.hp.com> From: "Mallikarjun C." To: , References: Subject: Re: [rddp] Re: iSCSI/iWARP drafts and flow control Date: Thu, 31 Jul 2003 17:43:45 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit > No new wire protocol is required. Please explain to me how credit can be replenished in the following example - without new positive flow control protocol. An initiator sends an immediate command in an untagged message to the target (and thus consumes one "fringe" credit). Considerations: a) CmdSN is not unique, can't be acked. b) target may discard it, or may process it c) no assumptions can be made based on subsequent command processing. The other specific iSCSI PDU types that I listed present a similar "problem" - for both directions. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Hewlett-Packard MS 5668 Roseville CA 95747 cbm@rose.hp.com ----- Original Message ----- From: "Caitlin Bestler" To: "Mallikarjun C." Cc: ; Sent: Wednesday, July 30, 2003 6:10 PM Subject: [rddp] Re: iSCSI/iWARP drafts and flow control > > On Wednesday, July 30, 2003, at 07:49 PM, Mallikarjun C. wrote: > > > > >> There is an identical flow control issue for > >> RDMA Reads. > > > > Not true. There's a built-in credit renewal in an RDMA Read. > > The Peer issuing the RDMA Read knows it can reuse the > > Read credit when it receives an RDMA Read Response. > > Send Messages carrying the "fringe" iSCSI PDUs need both > > new wire protocol and cross-layer chit-chat within an end-node > > between iSCSI and iSER - in order to renew credits. > > No new wire protocol is required. > > Restoration of no-CmdSNs credits could be piggy-backed on top > of the existing CmdSN credit system. > > Basically each Cmd reply restores one "Cmd Credit". It could > also restore an implicit number of "NoCmd" credits. Basically > if Cmd X had been proceeded by Y "NoCmd" messages, then the > reply to Cmd X would restore Y "NoCmd" credits. > > It is simple credit counting, suitable for implementation in > hardware or software at any number of layers, and requires > no additional wire messages or even fields. > > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 31 20:54:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20353 for ; Thu, 31 Jul 2003 20:54:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iOBH-00050K-Iy for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 20:54:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h710s39s019228 for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 20:54:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iOBH-000503-Ex for rddp-web-archive@optimus.ietf.org; Thu, 31 Jul 2003 20:54:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20337 for ; Thu, 31 Jul 2003 20:53:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iOBF-00017m-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 20:54:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19iOBE-00017j-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 20:54:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iOBF-0004z4-QU; Thu, 31 Jul 2003 20:54:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iOBA-0004yk-LK for rddp@optimus.ietf.org; Thu, 31 Jul 2003 20:53:56 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA20334 for ; Thu, 31 Jul 2003 20:53:52 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iOB8-00017g-00 for rddp@ietf.org; Thu, 31 Jul 2003 20:53:54 -0400 Received: from palrel13.hp.com ([156.153.255.238]) by ietf-mx with esmtp (Exim 4.12) id 19iOB7-00017d-00 for rddp@ietf.org; Thu, 31 Jul 2003 20:53:53 -0400 Received: from rosemail.rose.hp.com (rosemail.rose.hp.com [15.96.64.26]) by palrel13.hp.com (Postfix) with ESMTP id B7FE91C01762; Thu, 31 Jul 2003 17:53:52 -0700 (PDT) Received: from swathi (dhcp-roseor-bdj047.rose.hp.com [15.23.139.47]) by rosemail.rose.hp.com with SMTP (8.7.1/8.7.3 SMKit7.02) id RAA03386; Thu, 31 Jul 2003 17:53:51 -0700 (PDT) Message-ID: <021a01c357c7$5fbdab80$18a4f40f@rose.hp.com> From: "Mallikarjun C." To: "Kanevsky, Arkady" , , References: Subject: Re: [rddp] WG chair on buffer exhaustion (again) Date: Thu, 31 Jul 2003 17:53:50 -0700 MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 7bit X-Priority: 3 X-MSMail-Priority: Normal X-Mailer: Microsoft Outlook Express 6.00.2800.1106 X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2800.1106 Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Arkady, Please note the "....MUST however ensure that the affected DDP Segment is eventually recovered ....way that guarantees ULP-transparency" language in my suggested text. So the proposed text does not change anything wrt RDMAP and thus the automatic STag invalidation, and of course the definition of reliable connection stays the same as it was because the delivery semantics are still in-order. Regards. -- Mallikarjun Mallikarjun Chadalapaka Networked Storage Architecture Network Storage Solutions Hewlett-Packard MS 5668 Roseville CA 95747 cbm@rose.hp.com ----- Original Message ----- From: "Kanevsky, Arkady" To: ; Sent: Thursday, July 31, 2003 8:37 AM Subject: RE: [rddp] WG chair on buffer exhaustion (again) > David, > I am trying to comprehand the 2 issues. > > You stated that Mallikarjun's issue involves statistical (under-) > provisioning of buffers can happen without any Shared > Receive queues. > 1. What is the definition of reliable connection based on this change? > 2. How does UPL knows/control what is the connection behavior: > connection broken(termination DDP stream) if no recv buffer is > availble vs. > Send message dropped on the floor and not delivered, but DDP > connection stays? > 3. How is the Stag invalidation effected (if at all) with both proposals? > > Thanks, > Arkady > > Arkady Kanevsky email: arkady@netapp.com > Network Appliance phone: 781-768-5395 > 375 Totten Pond Rd. Fax: 781-895-1195 > 3rd Floor http: www.netapp.com > Waltham, MA 02451-2010 general phone: 781-768-5300 > > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Wednesday, July 30, 2003 5:40 PM > > To: rddp@ietf.org > > Subject: [rddp] WG chair on buffer exhaustion (again) > > > > > > With WG Chair hat on, let me step into this debate: > > > > > > Section 3.2 to: > > > > " Note, however, that the ULP needs to address flow control > > > > issues. > > Any > > > > ULP that runs on DDP SHOULD consider the lack of an > > associated > > receive > > > > buffer to place an inbound Untagged Message of the ULP as an > > exception." > > > > > > If the local interface has allows the DDP endpoint to know > > the valid > > > range of MSNs on a stream (which it will for simple Receive > > Queues at > > > the minimum) then it definitely SHOULD NOT consider the lack of a > > > buffer for an *invalid message* to be "an exception". > > > > Two issues are getting conflated here, and need to be kept separate: > > - Mallikarjun's issue involves statistical (under-) > > provisioning of buffers, > > so that an otherwise valid untagged message shows up > > without a buffer > > available to receive it (if the implementation had done > > worst-case > > provisioning, there would have been a buffer > > available). I believe > > there is WG rough consensus that this is an exceptional > > condition > > that does not need to immediately cause the termination > > of the DDP > > stream that sent the message, but it SHOULD not occur > > frequently. > > Mallikarjun's proposed text is close enough to what's > > needed for this. > > This condition could happen even on a simple Receive Queue if > > statistical (under-) provisioning is being used. > > - Caitlin's issue involves a peer violating a fixed > > per-DDP-stream limit on > > the number of untagged messages allowed . At the > > moment, the exact > > need and nature of such limits is still an open issue > > that needs to be > > resolved in the context of the security draft. When we > > have it resolved > > there, additional text will (probably) get driven into > > the DDP draft. > > If the issue is resolved in the manner Caitlin advocates, then > > her proposed text is certainly a candidate for what the > > DDP draft > > will need to say, but at the moment, the issue is open, and the > > DDP draft should not change what it says about this issue until > > it is resolved in the context of the security draft. > > > > Also, I've seen some confusing references to iSER and > > assumptions about local interface and resource sharing model. > > Shared Receive Queues have > > *not* been approved/adopted by this WG (and can't be until we > > get the resource sharing issue [2nd bullet above] resolved). > > To the extent that iSER requires Shared Receive Queues and > > can't cope with any other local resource model, it's probably > > made an overly restrictive design assumption that it's > > designers would be well-advised to reconsider, keeping in > > mind that the Hilland verbs draft is at best an *example* of > > a local interface to RDDP; it should not be viewed as > > normative functional requirements on that local interface. > > > > With luck, that's useful clarification, but feel free to ask > > if anything above doesn't make sense ... > > > > Thanks, > > --David > > ---------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 176 South St., Hopkinton, MA 01748 > > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > ---------------------------------------------------- > > > > _______________________________________________ > > rddp mailing list > > rddp@ietf.org > > https://www1.ietf.org/mailman/listinfo/rddp > > > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Jul 31 21:06:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA20670 for ; Thu, 31 Jul 2003 21:06:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iOMu-0005Fa-BL for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 21:06:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h71164Q4020176 for rddp-archive@odin.ietf.org; Thu, 31 Jul 2003 21:06:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iOMu-0005FL-8A for rddp-web-archive@optimus.ietf.org; Thu, 31 Jul 2003 21:06:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA20650 for ; Thu, 31 Jul 2003 21:05:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iOMr-0001CH-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 21:06:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19iOMr-0001CE-00 for rddp-web-archive@ietf.org; Thu, 31 Jul 2003 21:06:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iOMr-0005Ep-Di; Thu, 31 Jul 2003 21:06:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iOMC-0005EQ-Fm for rddp@optimus.ietf.org; Thu, 31 Jul 2003 21:05:20 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA20632 for ; Thu, 31 Jul 2003 21:05:16 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iOM9-0001Bw-00 for rddp@ietf.org; Thu, 31 Jul 2003 21:05:17 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19iOM9-0001Bn-00 for rddp@ietf.org; Thu, 31 Jul 2003 21:05:17 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 4A44C244BBF; Thu, 31 Jul 2003 21:04:58 -0400 (EDT) Date: Thu, 31 Jul 2003 20:04:45 -0500 Subject: Re: [rddp] Re: iSCSI/iWARP drafts and flow control Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: , To: "Mallikarjun C." From: Caitlin Bestler In-Reply-To: <020c01c357c6$0a21e340$18a4f40f@rose.hp.com> Message-Id: <23D23E30-C3BC-11D7-9DE7-003065D48EE0@asomi.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Thursday, July 31, 2003, at 07:43 PM, Mallikarjun C. wrote: > >> No new wire protocol is required. > > Please explain to me how credit can be replenished > in the following example - without new positive flow > control protocol. > > An initiator sends an immediate command in an untagged > message to the target (and thus consumes one "fringe" credit). > > Considerations: a) CmdSN is not unique, can't be acked. > b) target may discard it, or may process it c) no assumptions > can be made based on subsequent command processing. > > The other specific iSCSI PDU types that I listed present > a similar "problem" - for both directions. There are three cases: -- If there is no need to send a further untagged message then there is no need to reclaim the credit. -- If there is a later need to send a untagged message against the "main" credits, then the reply to that command will replenish the credits. -- Only exception: when sending an untagged message that requires the last "fringe credit", or when the "main" credits have been exhausted, a flag is set requesting an explicit flow control ack. Or a flag in the untagged message could request such a flow control ack. In general, responses to the "main" untagged messages replenish both "main" and "fringe" credits. This can be done without complex accounting, something along the lines of a "leaky bucket" algorithm. If you somehow exceed the "maximum" number of "fringe" credits without using "main" credits, you an simply use some form of "nop" command, or have a special "fringe" command to replenish "fringe" credits. But if such a a command is actually needed, it would mean that the sender is sending more "fringe" methods than the statistical methods would have allowed for in any event. Credits do not have to be run at a "as close to zero" basis as possible in order to qualify as flow control. You will probably object to having to send a reverse message "just" to ack the fringe methods. However, exactly such a message is being generated now at by TCP. If you really have a sequence of "fringe" / "oneway" messages, you are already generating a TCP segment to ack them. The only difference is that iSER would supply some data with that reverse flow TCP segment. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp