From exim@www1.ietf.org Fri Dec 5 20:35:26 2003 Received: from optimus.ietf.org ([132.151.1.19]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA06248 for ; Fri, 5 Dec 2003 20:35:25 -0500 (EST) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1ASRLh-00061x-IX for rddp-archive@odin.ietf.org; Fri, 05 Dec 2003 20:35:10 -0500 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id hB61Z9OE023177 for rddp-archive@odin.ietf.org; Fri, 5 Dec 2003 20:35:09 -0500 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1ASRLh-00061k-Ev for rddp-web-archive@optimus.ietf.org; Fri, 05 Dec 2003 20:35:09 -0500 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA06211 for ; Fri, 5 Dec 2003 20:34:54 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ASRLf-0005oJ-00 for rddp-web-archive@ietf.org; Fri, 05 Dec 2003 20:35:07 -0500 Received: from [132.151.1.19] (helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 1ASRLe-0005oF-00 for rddp-web-archive@ietf.org; Fri, 05 Dec 2003 20:35:06 -0500 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1ASRLb-00060j-J6; Fri, 05 Dec 2003 20:35:03 -0500 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1ASRKv-0005z7-OQ for rddp@optimus.ietf.org; Fri, 05 Dec 2003 20:34:22 -0500 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA06167 for ; Fri, 5 Dec 2003 20:34:06 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ASRKt-0005ne-00 for rddp@ietf.org; Fri, 05 Dec 2003 20:34:19 -0500 Received: from mail2.microsoft.com ([131.107.3.124]) by ietf-mx with esmtp (Exim 4.12) id 1ASRKs-0005mo-00 for rddp@ietf.org; Fri, 05 Dec 2003 20:34:19 -0500 Received: from mail6.microsoft.com ([157.54.6.196]) by mail2.microsoft.com with Microsoft SMTPSVC(6.0.3790.1041); Fri, 5 Dec 2003 17:33:49 -0800 Received: from inet-vrs-06.redmond.corp.microsoft.com ([157.54.6.181]) by mail6.microsoft.com with Microsoft SMTPSVC(6.0.3790.1069); Fri, 5 Dec 2003 17:33:28 -0800 Received: from 157.54.8.155 by inet-vrs-06.redmond.corp.microsoft.com (InterScan E-Mail VirusWall NT); Fri, 05 Dec 2003 17:33:46 -0800 Received: from red-imc-02.redmond.corp.microsoft.com ([157.54.9.107]) by inet-hub-04.redmond.corp.microsoft.com with Microsoft SMTPSVC(5.0.2195.6713); Fri, 5 Dec 2003 17:33:47 -0800 Received: from win-imc-01.wingroup.windeploy.ntdev.microsoft.com ([157.54.0.39]) by red-imc-02.redmond.corp.microsoft.com with Microsoft SMTPSVC(6.0.3790.0); Fri, 5 Dec 2003 17:35:40 -0800 Received: from WIN-MSG-10.wingroup.windeploy.ntdev.microsoft.com ([157.54.12.81]) by win-imc-01.wingroup.windeploy.ntdev.microsoft.com with Microsoft SMTPSVC(6.0.3790.1069); Fri, 5 Dec 2003 17:33:46 -0800 X-MimeOLE: Produced By Microsoft Exchange V6.5.7122.0 Content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [rddp] DDP 7.3 - Data Source MUST transmit DDP Messages in order Date: Fri, 5 Dec 2003 17:33:46 -0800 Message-ID: Thread-Topic: [rddp] DDP 7.3 - Data Source MUST transmit DDP Messages in order thread-index: AcO1IPMpJc49BXtlTLalmkPystOF1gGdU0xg From: "Jim Pinkerton" To: "Serag GadelRab" , "Caitlin Bestler" , Cc: "Michael Krause" X-OriginalArrivalTime: 06 Dec 2003 01:33:46.0087 (UTC) FILETIME=[FD9D3F70:01C3BB98] Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable I agree that state explosion on the receiver is a big concern.=20 It's really a question of requiring a scoreboarding receiver verses a single counter. Pretty radically different complexity, when you throw in that a message can be 2 gigabytes in length and segmented on any arbitrary boundary. Look at SACK, for example - they understood that state explosion is a huge deal, and at some point you have to punt. Their form of punting was capping the number and then allowing even that amount to be discarded if things really hit the fan. Note I'm _not_ suggesting that we implement SACK style limits on out-of-order processing. If we were to change, I'd personally prefer going the other direction - require in-order DDP segments on an inorder TCP stream. I think the issue you have is section 7.3, with the SHOULD verses having a MUST, specifically: At the Data Source, DDP ... * SHOULD transmit DDP Segments within a DDP Message in increasing MO order for Untagged DDP Messages and in increasing TO order for Tagged DDP Messages.=20 So hopefully the SHOULD means that the normal/fast path is in-order DDP Segments within an in-order TCP stream, but it also means that the receiver must be able to handle the out-of-order DDP segments within a DDP message on an in-order TCP stream case. So it doesn't get you out of the state explosion - it just means that you don't have to necessarily handle it in the fast path. Jim > -----Original Message----- > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org] On Behalf Of Serag > GadelRab > Sent: Thursday, November 27, 2003 11:53 AM > To: 'Caitlin Bestler'; rddp@ietf.org > Cc: 'Michael Krause' > Subject: RE: [rddp] DDP 7.3 - Data Source MUST transmit DDP Messages in > order >=20 >=20 > From an implementation point of view, handling out-of-order DDP segments > within in-order TCP flow is not going to come for free. Special HW > will have to be added to ensure that this case is handled properly since > it does not correspond exactly to the case where the TCP segments arrive > out of order. >=20 > The question is whether this added complexity in receivers is warranted. > I doubt that any transmitters will try to engage in such optimizations > due to the complexity of scheduling OOO DDP segments. Moreover, one > could argue that this type of optimization is not necessary since the > data that needs to be transmitted will almost certainly be resident in > host memory when the transmit WQE is generated to the RNIC; so why > burden receivers with such a functionality when it will probably be > never exhibited by a transmitter? >=20 > Serag >=20 >=20 >=20 > -----Original Message----- > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org] On Behalf Of > Caitlin Bestler > Sent: Thursday, November 06, 2003 11:11 AM > To: rddp@ietf.org > Cc: Michael Krause > Subject: RE: [rddp] DDP 7.3 - Data Source MUST transmit DDP Messages in > order >=20 >=20 >=20 > Michael Krause's replied to my post as follows: >=20 > >> I would request the following: > >> > >> - Does the WG agree that the current wording is compatible > >> with both interpretations? Or is the latter interpretation > >> locked in by some other language the at I overlooked? >=20 > >> - If there is agreement that the current language is at > >> least potentially ambiguous, is there any objection to > >> adding the clarification that all DDP Segments for a > >> DDP Message must be submitted for transmission to the > >> LLP before any Segment for a following DDP Message. >=20 > > Implementations should be able to deal with OOO packet delivery and > being > > able to target multiple receive buffers if they are posted (shared or >=20 > exclusive). > > Given there is no delta in the wire protocol or the expected > interoperability in > > this respect, I do not favor trying to tighten how the transmitter > is implemented > > as that is potentially delving into the specific design options in > the hardware. > > > > Mike >=20 > Generally, I am in full agreement. Receivers should be as flexible as > possible > and senders should be given as much freedom as possible. >=20 > But the purpose of that is to allow senders to find potential > optimizations > in ways that might not have been obvious. For example, mandating that a > message MUST be segmented in a certain way would block a sender from > optimizing > for a DDP Segment that might have piggy-packed in the same SCTP Packet > or TCP > Segment that was already partially filled. >=20 > But I'm not sure I can see any benefit to the senders of working ahead, > especially when the last segments must be sent in order and when for > many > ULPs almost all untagged messages will fit in a single segment. >=20 > On the receiving side, however, the restriction can be beneficial to an > entire *class* of implementations -- those that ride on top of optimized > TCP solutions. >=20 > Since we have made it clear that the receiver does not have to > pre-commit > receive buffer for untagged messages (as in the class receive queue) we > have implicitly allowed for receivers that must fetch buffers "on > demand". >=20 > Whatever implementation is used, clearly having a limit on the number of > fetched buffers that have not completed will result in a simpler > problem. >=20 > Simplifying the problem is legitimate. Yes it benefits certain > implementations, > but that is because it simplified the problem. >=20 > Admittedly, the proposed restriction is of virtually no benefit to > implementations that process DDP Segments in arrival order. For that > reason it would be wrong to impose any costs on those implementations > for this feature. But there is no impact on out-of-order receivers. >=20 > The *only* impact is that it might block hyper-aggressive re-ordering of > unsent pending DDP Segments. Even with this proposed additional > language, such re-ordering is still possible *within* an untagged > message and with tagged segments. In my opinion that is more than enough > slack for potential optimizations (in fact I suspect that very few > implementations are even considering such optimizations). >=20 >=20 >=20 >=20 >=20 >=20 > -- > Caitlin Bestler - cait@asomi.com - http://asomi.com/ > http://asomi.com/CaitlinBestlerPublicPgpKey.html >=20 >=20 > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp >=20 >=20 > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Dec 8 18:25:21 2003 Received: from optimus.ietf.org ([132.151.1.19]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA11999 for ; Mon, 8 Dec 2003 18:25:21 -0500 (EST) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1ATUkW-00037Y-5Q for rddp-archive@odin.ietf.org; Mon, 08 Dec 2003 18:25:08 -0500 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id hB8NP85a011996 for rddp-archive@odin.ietf.org; Mon, 8 Dec 2003 18:25:08 -0500 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1ATUkV-00037P-Of for rddp-web-archive@optimus.ietf.org; Mon, 08 Dec 2003 18:25:07 -0500 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA11965 for ; Mon, 8 Dec 2003 18:24:49 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ATUkS-0001SD-00 for rddp-web-archive@ietf.org; Mon, 08 Dec 2003 18:25:04 -0500 Received: from manatick.foretec.com ([4.17.168.5] helo=manatick) by ietf-mx with esmtp (Exim 4.12) id 1ATUkS-0001S9-00 for rddp-web-archive@ietf.org; Mon, 08 Dec 2003 18:25:04 -0500 Received: from [132.151.6.22] (helo=optimus.ietf.org) by manatick with esmtp (Exim 4.24) id 1ATUkb-0007Ua-02 for rddp-web-archive@ietf.org; Mon, 08 Dec 2003 18:25:13 -0500 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1ATUkR-00036l-KI; Mon, 08 Dec 2003 18:25:03 -0500 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1ATUje-00035g-VQ for rddp@optimus.ietf.org; Mon, 08 Dec 2003 18:24:14 -0500 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA11905 for ; Mon, 8 Dec 2003 18:23:57 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1ATUjc-0001R6-00 for rddp@ietf.org; Mon, 08 Dec 2003 18:24:12 -0500 Received: from palrel11.hp.com ([156.153.255.246]) by ietf-mx with esmtp (Exim 4.12) id 1ATUja-0001Qx-00 for rddp@ietf.org; Mon, 08 Dec 2003 18:24:10 -0500 Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel11.hp.com (Postfix) with ESMTP id F0E101C00351 for ; Mon, 8 Dec 2003 15:24:05 -0800 (PST) Received: from mk731915.cup.hp.com ([15.244.170.65]) by esmail.cup.hp.com (8.11.1/8.8.6) with ESMTP id hB8NFsi27960 for ; Mon, 8 Dec 2003 15:15:54 -0800 (PST) Message-Id: <5.1.0.14.2.20031208152216.05d55e48@15.13.130.73> X-Sender: krause@15.13.130.73 X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Mon, 08 Dec 2003 15:22:39 -0800 To: rddp@ietf.org From: Michael Krause Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: [rddp] Security Draft Comments Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Overall, the security draft is coming together with no real major issues to resolve from what I've reviewed to date thus we should be able to get it cleaned up in a relatively short period of time. Mike (1) Draft might benefit from a section that describes current RDMA / DDP techniques that limit exposures. For example, a description of the basic error semantics and how these lead to hard errors and connection shutdown. While some of this is implied or spread out to varying extents within the draft specification, consolidation into a single section would simplify the subsequent discussion as well as add clarity. Applicable error conditions to document: protection violation, STag violation, base / bounds violation, etc. In addition, a brief description of window vs. region and how the fine grain exposure of a window has benefits to the ULP in constraining potential memory to target. (2) Unclear if there is a need for a glossary but some specification terms are repeated here while others are not. May be simpler to just reference the specs with the agreed to terminology and definitions rather than redefine in potentially imprecise terms. (3) Figure 1. One might consider having a line between the admin and the privileged application given the resource manager is not required but its essence may be integrated into the privileged application. For example, a storage subsystem may already have defined interfaces to administration services and if it has exclusive control of the RNIC would not require a separate resource manager. (4) The draft often repeats information making the flow / structure somewhat awkward. It would be better to have a given interface, definition, etc. defined once with all relevant material than to spread the material out and thus repeat it throughout the document. (5) Section 4 is a bit rough in its flow / structure. Subsections could be simplified into a somewhat standard structure. This builds upon the previous comment. Perhaps each component could be: - Defined - List of responsibilities as they relate to RDMA / DDP - List of resources and what is local and what is visible from the remote - Is the resource shared or not - How each component is initialized - List of potential issues w.r.t. security exposures / attacks / risks, i.e. threats - List of techniques to eliminate / reduce security threats This would provide more focus on each component given the material is consolidated. I tend to prefer such an approach because it helps bring together all of the relevant information required to define and protect a resource making it more likely developers will get it right. (6) Request proxy interface definition in 4.1 really does not provide much information. It provides a high-level statement on counter measures but due to organization, such a statement does not have much in terms of value. (7) 4.2.4 Is there any need to discuss the 8-bit key portion of the STag and how an application / ULP can help control the wire-view of the STag? (8) 4.28 There is "Event Queue" and one other line but not really anything else stated here. This seems almost like a fragment or a TBD area. (9) 4.3 Is 4.3 just a place holder? Not clear its relevance to the draft as many system properties can be attacked but these are outside the scope of RDMA itself. (10) 5.0 The pros / cons of sharing resources should be focused on security issues. Any application / subsystem that does not share obviously can be limited or denied requested resources - this is a standard problem for an OS / policy manager to resolve. It is simpler to just state that the system operates in one of two modes: - Use of a resource policy manager that dynamically determines what, if any, resources are provided and the amount of resources. How dynamic resource management is an implementation-specific problem and outside the scope of the draft specification. - Fixed resource assignment to a given application. The application knows what its limits are through whatever means (could be an embedded environment thus there is no administration interface). Some of this is covered in various areas but consolidating this all into one section focused on "to share or share not, that is the question?" would simplify the document. (11) It would be good to have consistent usage of "privileged" when describing the resource manager. (12) Page 15 under NS-RT. There is a note attached but not really sufficient information provided w.r.t. the key-based authentication per packet sentence. Either provide more information here or add a pointer to the appropriate specification or subsequent section where this is described further. (13) 7.1.1. Add "each" to , it is recommend that Protection Domain be limited to the number.... (14) 7.1.2 Second bullet should use the term window or cross reference to the appropriate specification so that the reader can understand how this is somewhat a local issue and that there are tools defined to help provide this service. (15) 7.1.2 fourth bullet, STag needs to be properly capitalized. It would also be good to cross reference back to the error description (per previous comment) the effects of a STag violation. (16) 7.1.5 Need to emphasize that this is an application problem not a RDMA or DDP problem. (17) 7.2 My reaction was "so". Not clear why this section is there given the limited information provided. (18) 7.2.1 Need to think about this more but it seems there could be nasty middle-box attacks which are outside the scope of RDMA and DDP. It would also be good to separate authentication from integrity protection as the two provide different protections and the associated protections should be clearly attributed to each if you are going to list out the various attacks. (19) 7.2.2 Repeats earlier information on RDMA Read attack. The spec might benefit from simply having a set of ladder diagrams of operations for different sequences with a list of associated attack and how one provides countermeasures. (20) 7.2.2. STag of zero is a local, not remote value and thus should not be part of the allocation algorithm. (21) 7.3.2 This may be a bit of implementation but it may be possible that a RNIC could have multiple execution paths such that the invalidate executes in parallel or serial to other operations thus a subsequent RDMA operation could access memory prior to the invalidate being completed. Same applies if the invalidate or access rights changes are implemented in software which is asynchronous the subsequent hardware packet processing operations. This might lead to a race condition and thus exposure. One question: What if the remote stated it would invalidate but does not do so? It is clear that remote peers should not be trusted but one may appear to be fully compliant with the ULP needs and then simply not invalidate thus enabling the memory to be read / updated. This can be dealt with via the implementation - a couple of simple techniques come to mind - but it may need to be discussed a bit further. (22) 7.4.2 Zeroing of a data buffer or providing valid data with all non-used buffer memory being invalidated / zero should be noted. (23) 7.5.2.2 Note that this is only for a shared CQ in the header though the subsequent header does make this more specific. (24) 7.5.3 It is not clear to me that a RDMA MIB is required. One can understand whether a LLP or a physical interface are making forward progress through the existing MIBs. Providing additional statistics isn't really of value as one knows the connections / streams and layer 2 are making forward progress. After that, one checks to see if the ULP is making forward progress and therefore we have a complete picture without requiring yet another MIB to implement / manage. (25) Not sure if I agree with the note in 7.5.3 that indicates that there isn't risk for some resources. For example, a ULP may negotiate use of large receive buffers and potentially large number of receive buffers and then not make forward progress. These types of consumptions are problems for both local and remote attacks. (26) 7.5.4 One could also see an attack where one purposely transmits out of order packets or delays the packet retransmission quite often. This could force some implementations to take the slow path quite often thus causing a performance degradation attack. This is an implementation-specific problem that is also applicable to non-RDMA network stacks so it isn't necessarily new but might be beneficial to note. (27) 7.5.5 The last sentence says that the remote peer just can't use the STag but it can also cause a connection to be shut down on error thus while true, there is another potential consequence. (28) 7.5.6 Should note the high and low-water marks per connection sharing a receive queue to detect and allow corrective action to take place. (29) 7.6 Not sure if this applies but if the system supports dynamic patching of a subsystem, then one might replace the executable with a trojan horse thus enabling one to take over the system / resources. Might be worthwhile to have a note that such patching solutions should have all RDMA / DDP resources released so that the LLP establishment process is required thereby providing some level of new authentication to occur to the previous remote peers. (30) Most of sections 8 and 9 are works in progress so no additional comments made. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Sun Dec 21 10:17:31 2003 Received: from optimus.ietf.org ([132.151.1.19]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA26758 for ; Sun, 21 Dec 2003 10:17:30 -0500 (EST) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1AY5KJ-0001pr-Ri for rddp-archive@odin.ietf.org; Sun, 21 Dec 2003 10:17:04 -0500 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id hBLFH32w007051 for rddp-archive@odin.ietf.org; Sun, 21 Dec 2003 10:17:03 -0500 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1AY5KJ-0001pe-Mo for rddp-web-archive@optimus.ietf.org; Sun, 21 Dec 2003 10:17:03 -0500 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA26737 for ; Sun, 21 Dec 2003 10:17:00 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AY5KH-0000FB-00 for rddp-web-archive@ietf.org; Sun, 21 Dec 2003 10:17:01 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AY5KF-0000F2-00 for rddp-web-archive@ietf.org; Sun, 21 Dec 2003 10:17:00 -0500 Received: from [132.151.1.19] (helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 1AY5KF-0000Ex-00 for rddp-web-archive@ietf.org; Sun, 21 Dec 2003 10:16:59 -0500 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1AY5KF-0001oW-QZ; Sun, 21 Dec 2003 10:16:59 -0500 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1AY5Ji-0001o8-UR for rddp@optimus.ietf.org; Sun, 21 Dec 2003 10:16:26 -0500 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA26686 for ; Sun, 21 Dec 2003 10:16:23 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AY5Jg-0000E1-00 for rddp@ietf.org; Sun, 21 Dec 2003 10:16:24 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AY5Jf-0000Dt-00 for rddp@ietf.org; Sun, 21 Dec 2003 10:16:24 -0500 Received: from mx01.netapp.com ([198.95.226.53]) by ietf-mx with esmtp (Exim 4.12) id 1AY5Je-0000DC-00; Sun, 21 Dec 2003 10:16:23 -0500 Received: from hawk.corp.netapp.com (hawk [10.57.156.122]) by mx01.netapp.com (8.12.10/8.12.10/NTAP-1.4) with ESMTP id hBLFFqKw025437; Sun, 21 Dec 2003 07:15:52 -0800 (PST) Received: from svlexc01.hq.netapp.com (svlexc01.corp.netapp.com [10.10.22.171]) by hawk.corp.netapp.com (8.12.9/8.12.9/NTAP-1.5) with ESMTP id hBLFFqGV003138; Sun, 21 Dec 2003 07:15:52 -0800 (PST) Received: from tmt.netapp.com ([10.97.6.30]) by silver.hq.netapp.com with Microsoft SMTPSVC(5.0.2195.5329); Sun, 21 Dec 2003 10:15:49 -0500 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C3C7D5.508EC080" content-class: urn:content-classes:message X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 Date: Sun, 21 Dec 2003 07:14:23 -0800 Message-ID: <5.2.1.1.2.20031221094643.021237a0@silver.nane.netapp.com> Thread-Topic: NFS/RDMA ONC RPC reference implementation release Thread-Index: AcPH1VEsTwogVuddQeOpx+HvSMepEw== From: "Talpey, Thomas" To: Cc: Subject: [rddp] NFS/RDMA ONC RPC reference implementation release Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.5 required=5.0 tests=HTML_20_30,HTML_MESSAGE autolearn=no version=2.60 This is a multi-part message in MIME format. ------_=_NextPart_001_01C3C7D5.508EC080 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Network Appliance is pleased to announce general availability of a client implementation of NFSv3 over RDMA for Linux kernels. This client, with full source code, is available for download from http://sourceforge.net/projects/nfs-rdma/ The client is implemented entirely within the Linux ONC RPC code, and requires no changes to the NFS layer. It is a relatively complete implementation of the draft ONC RPC RDMA protocols published earlier this year. NetApp is publishing this implementation in order to encourage interest and support of this protocol, and to enable the future use of NFSv4 over RDMA. The Linux v3 client will be tested at the upcoming NFS Connectathon in February. The project description with references is included below. We look = forward to discussion and experimentation with the new protocol in the months to come. Please see the above Sourceforge project for updates. Happy New Year! Tom. ----- Project Description: http://sourceforge.net/projects/nfs-rdma/ The NFS/RDMA project is a reference implementation of a new RDMA-capable ONC RPC transport for use by Linux kernel NFS.=20 Client support for NFSv3 is the initial goal. This transport = implementation is available under a BSD-style license.=20 Through use of an RDMA transport, a very high level of performance may = be achieved, including zero-copy and zero-touch direct I/O. No = modifications to the Linux NFSv3 client code are required.=20 The protocol is conformant to that published in the following Internet = Drafts:=20 draft-callaghan-rpcrdma-00.txt=20 draft-callaghan-nfsdirect-00.txt=20 draft-talpey-nfs-rdma-problem-statement-00.txt=20 and the implementation uses the kDAPL RDMA API=20 http://sourceforge.net/projects/dapl/=20 The kDAPL API supports Infiniband and iWARP RDMA transports.=20 Followon goals include server support, NFSv4 support, and additional = transport research.=20 The Linux NFS kernel server code is not addressed in the initial = release. Portions of the transport implementation may provide the base for future server = work. NFSv4 support may incorporate protocol modifications from the Internet = Draft:=20 draft-talpey-nfsv4-rdma-sessions-00.txt=20 An RDMA-enabled NFSv4 implementation is a natural high performance filesystem for clustered or workgroup computing, due to the enhanced = file sharing capabilities of NFSv4, such as its integrated locking, = delegations, and security.=20 The transport interface framework created as part of this project is = general, and may be applied to other efforts. Support for RPC transports = employing TCP Offload Engines, or hybrid approaches using Fibre Channel/iSCSI in conjunction with NFS will also be considered. ------_=_NextPart_001_01C3C7D5.508EC080 Content-Type: text/html; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable NFS/RDMA ONC RPC reference implementation release

Network Appliance is pleased to announce general = availability of a
client implementation of NFSv3 over RDMA for Linux = kernels. This
client, with full source code, is available for = download from

        http://sourceforge.net= /projects/nfs-rdma/

The client is implemented entirely within the Linux = ONC RPC code,
and requires no changes to the NFS layer. It is a = relatively complete
implementation of the draft ONC RPC RDMA protocols = published
earlier this year.

NetApp is publishing this implementation in order to = encourage interest
and support of this protocol, and to enable the = future use of NFSv4 over
RDMA. The Linux v3 client will be tested at the = upcoming NFS
Connectathon in February.

The project description with references is included = below. We look forward
to discussion and experimentation with the new = protocol in the months to
come. Please see the above Sourceforge project for = updates.

Happy New Year!

Tom.

-----

Project Description:

http://sourceforge.net= /projects/nfs-rdma/

The NFS/RDMA project is a reference implementation of = a new RDMA-capable
ONC RPC transport for use by Linux kernel NFS. =

Client support for NFSv3 is the initial goal. This = transport implementation is
available under a BSD-style license.

Through use of an RDMA transport, a very high level of = performance may be
achieved, including zero-copy and zero-touch direct = I/O. No modifications to
the Linux NFSv3 client code are required.

The protocol is conformant to that published in the = following Internet Drafts:
        = draft-callaghan-rpcrdma-00.txt
        = draft-callaghan-nfsdirect-00.txt
        = draft-talpey-nfs-rdma-problem-statement-00.txt

and the implementation uses the kDAPL RDMA API
        http://sourceforge.net/pro= jects/dapl/

The kDAPL API supports Infiniband and iWARP RDMA = transports.

Followon goals include server support, NFSv4 support, = and additional transport
research.

The Linux NFS kernel server code is not addressed in = the initial release. Portions
of the transport implementation may provide the base = for future server work.

NFSv4 support may incorporate protocol modifications = from the Internet Draft:
        = draft-talpey-nfsv4-rdma-sessions-00.txt

An RDMA-enabled NFSv4 implementation is a natural high = performance
filesystem for clustered or workgroup computing, due = to the enhanced file
sharing capabilities of NFSv4,  such as its = integrated locking, delegations,
and security.

The transport interface framework created as part of = this project is general,
and may be applied to other efforts. Support for RPC = transports employing
TCP Offload Engines, or hybrid approaches using Fibre = Channel/iSCSI in
conjunction with NFS will also be considered.

------_=_NextPart_001_01C3C7D5.508EC080-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Dec 22 18:28:11 2003 Received: from optimus.ietf.org ([132.151.1.19]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA17393 for ; Mon, 22 Dec 2003 18:28:11 -0500 (EST) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1AYZSj-0006Da-78 for rddp-archive@odin.ietf.org; Mon, 22 Dec 2003 18:27:45 -0500 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id hBMNRjok023902 for rddp-archive@odin.ietf.org; Mon, 22 Dec 2003 18:27:45 -0500 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1AYZSj-0006DR-0F for rddp-web-archive@optimus.ietf.org; Mon, 22 Dec 2003 18:27:45 -0500 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA17356 for ; Mon, 22 Dec 2003 18:27:39 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AYZOF-0005gE-00 for rddp-web-archive@ietf.org; Mon, 22 Dec 2003 18:23:07 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AYZKH-0005Wj-00 for rddp-web-archive@ietf.org; Mon, 22 Dec 2003 18:19:07 -0500 Received: from [132.151.1.19] (helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 1AYZKG-0005WM-00 for rddp-web-archive@ietf.org; Mon, 22 Dec 2003 18:19:00 -0500 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1AYZKG-0005wx-Vm; Mon, 22 Dec 2003 18:19:00 -0500 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 1AYZK7-0005vW-SI for rddp@optimus.ietf.org; Mon, 22 Dec 2003 18:18:51 -0500 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA16884 for ; Mon, 22 Dec 2003 18:18:47 -0500 (EST) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 1AYZK4-0005S0-02 for rddp@ietf.org; Mon, 22 Dec 2003 18:18:49 -0500 Received: from exim by ietf-mx with spam-scanned (Exim 4.12) id 1AYZIe-0005PG-00 for rddp@ietf.org; Mon, 22 Dec 2003 18:17:20 -0500 Received: from mx01.netapp.com ([198.95.226.53]) by ietf-mx with esmtp (Exim 4.12) id 1AYZIe-0005P4-00 for rddp@ietf.org; Mon, 22 Dec 2003 18:17:20 -0500 Received: from frejya.corp.netapp.com (frejya [10.57.157.119]) by mx01.netapp.com (8.12.10/8.12.10/NTAP-1.4) with ESMTP id hBMNGmKw012229; Mon, 22 Dec 2003 15:16:48 -0800 (PST) Received: from svlexc01.hq.netapp.com (svlexc01.corp.netapp.com [10.10.22.171]) by frejya.corp.netapp.com (8.12.9/8.12.9/NTAP-1.5) with ESMTP id hBMNGlpv013951; Mon, 22 Dec 2003 15:16:48 -0800 (PST) content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 Date: Mon, 22 Dec 2003 15:16:40 -0800 Message-ID: Thread-Topic: Proposal to augment MPA connection set up frame Thread-Index: AcPI4abFqkwvbjuiSL2a0Gn7kHohPw== From: "Kanevsky, Arkady" To: Content-Transfer-Encoding: quoted-printable Subject: [rddp] Proposal to augment MPA connection set up frame Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , X-Spam-Checker-Version: SpamAssassin 2.60 (1.212-2003-09-23-exp) on ietf-mx.ietf.org X-Spam-Status: No, hits=0.0 required=5.0 tests=none autolearn=no version=2.60 Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable I have a comment on MPA draft. It now defines very nicely an Initiator/Responder sequence. But it looks deficient for a broad set of ULPs. Most of the client-server ULP uses two type of Responses: "accept" and "reject". The MPA draft does not provide a way to differentiate between these 2 responses. Here is one example of an application use. One of the common way the private data is used is to negotiate RDMA Read credits for a connection to be used by RDDP. An Initiator passes in private data a requested RDMA credits. Responder either accepts a connection or rejects a request specifying how many RDMA Read credits it can support. Currently the Responder has 2 legal choices. One it can terminate a connection. This does not convey any information to the Initiator. The second choice is to generate Responder Frame even though RDMA Read credits requested can not be satisfied. This will establish connection but it can not be used as intended by the Initiator. Initiator can either tear down the established stream mode connection or use connection with Responder RDMA credits supported. This is de-facto ULP protocol since Responder Frame will include the RDMA Read credits it can support. On top of that both sides have to agree on the action they will take. While ULP can use a protocol over private data to differentiate between accept and reject, given a commonality of this semantic for ULPs we can use one bit from the Reserved area of MPA frame for the Responder frame to differentiate between accept and reject responses. Proposal: for Responder MPA frame the 1st Reserved bit to be used as accept/reject bit. 0 for accept, 1 for reject. The bit will be called Type (T) in figure 7. Arkady Kanevsky email: arkady@netapp.com Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 3rd Floor http: www.netapp.com Waltham, MA 02451-2010 general phone: 781-768-5300 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp