From mailnull@www1.ietf.org Mon Mar 3 14:42:35 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA09479 for ; Mon, 3 Mar 2003 14:42:35 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h23Jqgs19953 for rddp-archive@odin.ietf.org; Mon, 3 Mar 2003 14:52:42 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h23Jqgp19950 for ; Mon, 3 Mar 2003 14:52:42 -0500 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA09464 for ; Mon, 3 Mar 2003 14:42:04 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h23JpOp19876; Mon, 3 Mar 2003 14:51:24 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h23JoIp19765 for ; Mon, 3 Mar 2003 14:50:18 -0500 Received: from emulex.emulex.com (emulex.emulex.com [138.239.112.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA09401 for ; Mon, 3 Mar 2003 14:39:41 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id LAA13077; Mon, 3 Mar 2003 11:40:42 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Mon, 3 Mar 2003 14:40:41 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B900C@xbl.ma.emulex.com> From: "Williams, Jim" To: "'Caitlin Bestler'" Cc: "'rddp@ietf.org'" Subject: RE: [rddp] SCTP mapping - Multiple Streams Date: Mon, 3 Mar 2003 14:40:40 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > >I am guessing that most implementations will chose to > >support one stream per association because it is simpler > >and I don't see a performance down side. However since > >number of streams per association is negotiable, it is > >fine to allow more. > > On the contrary, there are major advantages to sharing > the same association across multiple DDP streams > consolidates acking, congestion control and setup. The down side for the implementation is that two contexts (stream and association) must be accessed for each packet sent and received rather than one combined context. This is a factor of two "cost" increase in one of the more expensive aspects of protocol acceleration. As an implementer, I would not lightly incur this. I don't see the advantages as particularly compelling. > An integrated RDDP/SCTP implementation SHOULD NOT presume > there is only one stream per association unless it has > very strong reasons to believe that there will only be a > single stream between the two endpoints. My understanding is that one end of an SCTP association may specify one stream per association and the other end has no choice but to comply as the number is negotiated by the minimum of the two. So there is no problem in making such an presumption. The implementation is simpler if separate associations are used for each stream between endpoints. Whether an implementation chooses to do one stream per association or multiple streams per associations is a design choice that does not need to be mandated either way by the mapping standard, and is not mandated by the proposal. However I would prefer to see the number of streams per association layered opaquely below DDP so that the DDP layer does not know or care about which of its streams share or do not share an association. Doing otherwise unnecessarily complicates the layering model with no added value. Otherwise you end with a situation where moving from RDDP/TCP to RDDP/SCTP requires unnecessary application changes. This is not a good thing. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Mar 3 15:23:19 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA11602 for ; Mon, 3 Mar 2003 15:23:19 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h23KXRg22969 for rddp-archive@odin.ietf.org; Mon, 3 Mar 2003 15:33:27 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h23KXQp22966 for ; Mon, 3 Mar 2003 15:33:26 -0500 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA11596 for ; Mon, 3 Mar 2003 15:22:48 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h23KW2p22926; Mon, 3 Mar 2003 15:32:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h23KVtp22906 for ; Mon, 3 Mar 2003 15:31:55 -0500 Received: from thebe.your-site.com (ns5.your-site.com [140.186.45.30] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA11550 for ; Mon, 3 Mar 2003 15:21:16 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 250172451BA; Mon, 3 Mar 2003 15:22:41 -0500 (EST) Date: Mon, 3 Mar 2003 14:24:24 -0600 From: Caitlin Bestler Subject: RE: [rddp] SCTP mapping - Multiple Streams To: "Williams, Jim" Cc: "'rddp@ietf.org'" X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B900C@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 3/3/03, Williams, Jim wrote: >> >I am guessing that most implementations will chose to >> >support one stream per association because it is simpler >> >and I don't see a performance down side. However since >> >number of streams per association is negotiable, it is >> >fine to allow more. >> >> On the contrary, there are major advantages to sharing >> the same association across multiple DDP streams >> consolidates acking, congestion control and setup. > >The down side for the implementation is that two contexts >(stream and association) must be accessed for each >packet sent and received rather than one combined context. >This is a factor of two "cost" increase in one of the >more expensive aspects of protocol acceleration. As >an implementer, I would not lightly incur this. >I don't see the advantages as particularly compelling. > For an established association[1], the association is easily found with the verification tag. From there the source/destination addresses should be verified to filter out spoofed packets. That is true whether you have a "consolidated" context or not. >From there you need to perform per-stream processing. This is a simple lookup within the context of the already found association. This is the *only* step that having a one-association / one-stream pairing would save. Meanwhile it would avoid savings in association setup (a four-way handshake), Acks (the overhead of which could be shared with multiple streams) and congestion control (each stream would have to do its own slow start). [1] An established association is where the performance issues are most important. Matching an incoming SCTP packet during association setup and teardown is more complex, as is currently being discussed in the sctp-implementors list. >> An integrated RDDP/SCTP implementation SHOULD NOT >> presume there is only one stream per association unless >> it has very strong reasons to believe that there will >> only be a single stream between the two endpoints. > >My understanding is that one end of an SCTP association >may specify one stream per association and the other >end has no choice but to comply as the number is negotiated >by the minimum of the two. So there is no problem in >making such an presumption. The implementation is >simpler if separate associations are used for each >stream between endpoints. > Yes that would be allowed. But that presumes that the association creator knows that there will be no other streams. How would it know this? The key issue is who is the owner of the association? The ULP or the RDDP stack? If there are multiple clients that want to access the same remote service, SHOULD they use the same association? As I read the SCTP design, the applicability statement and the sockets API, I believe there is an intent that an association should be shared by multiple streams. One of the major comparison points versus TCP cited is that applications such as HTTP force applications to create redundant parallel connections, which is unfair and unproductive use of network resources. >Whether an implementation chooses to do one stream >per association or multiple streams per associations >is a design choice that does not need to be mandated >either way by the mapping standard, and is not mandated >by the proposal. > I agree that mandating is probably excessive, but I believe it ought to remain at the SHOULD level. One of the intents of SCTP is to consolidate the overhead created by parallel connections. RDDP's mapping should be consistent with this. >However I would prefer to see the number of streams >per association layered opaquely below DDP so that >the DDP layer does not know or care about which of its >streams share or do not share an association. >Doing otherwise unnecessarily complicates the layering >model with no added value. Otherwise you end with >a situation where moving from RDDP/TCP to RDDP/SCTP >requires unnecessary application changes. This is not >a good thing. That is a valid point. But one that can be solved by simply viewing the stream as an extension of the port number. The "connection qualifier" is the port number + stream number, where TCP only supports stream 0. This approach can be extremely valuable when the client machine uses multiple DDP streams to the same server. For example a web server accessing a file server via RDDP/SCTP. Traffic shaping can be performed for the total workload, whille each individual HTTP fetch is too short to be effectively traffic shaped. No per-fetch setup is required. And each external HTTP connection can use its own DDP stream to interact with the file server (avoiding the need for a central queuing of file server requests). In any event, I think you are correct that this issue cannot be properly discussed in the absence of some discussion of the local interface. I believe this requires a discussion of how a ULP would create a DDP Stream *without* knowledge of the underlying LLP. I would propose that the aforementioned "connection qualifier" is one very simple method of doing so. The "LLP Stream" is identified by the remote IP Address(es), the LLP protocol (SCTP or TCP), the port and the stream number. The stack would create the association if this was the first DDP stream requiring it. Later, it could be deleted when all of its streams had been terminated. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Mar 3 16:12:45 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13074 for ; Mon, 3 Mar 2003 16:12:45 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h23LMsa27170 for rddp-archive@odin.ietf.org; Mon, 3 Mar 2003 16:22:54 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h23LMsp27167 for ; Mon, 3 Mar 2003 16:22:54 -0500 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13048 for ; Mon, 3 Mar 2003 16:12:14 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h23LLip27093; Mon, 3 Mar 2003 16:21:44 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h23LKlp27048 for ; Mon, 3 Mar 2003 16:20:47 -0500 Received: from thebe.your-site.com (ns5.your-site.com [140.186.45.30] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13012 for ; Mon, 3 Mar 2003 16:10:07 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 29550245159 for ; Mon, 3 Mar 2003 16:11:39 -0500 (EST) Date: Mon, 3 Mar 2003 15:13:23 -0600 From: Caitlin Bestler Subject: RE: [rddp] SCTP mapping - Multiple Streams (binding) To: rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit To be a bit more specific, the local interface that would give control over SCTP Stream selection would be the equivalent of a 'bind'. If the ULP chooses to bind, the low 16-bits of the "port number" would specific the SCTP or TCP port number. The high (or next) 16-bits would specify the DDP Stream. An application that was unaware of multiple streams would bind Stream 0 by default, which even TCP supports. If the application did not issue a bind, which is typical, the DDP layer would choose the port and stream for it. The adaptation protocol already includes a reset-notice to inform the other side that a new DDP Stream has been initialized. This is one method. It would also be possible to use SLP or some IB-CM equivalent to select streams. The benefits of stream consolidation is not incompatible with maintaining source level TCP compatability for the ULP. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Mar 3 16:33:01 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13615 for ; Mon, 3 Mar 2003 16:33:01 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h23LhB528822 for rddp-archive@odin.ietf.org; Mon, 3 Mar 2003 16:43:11 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h23LhAp28819 for ; Mon, 3 Mar 2003 16:43:10 -0500 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13609 for ; Mon, 3 Mar 2003 16:32:30 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h23Lg1p28715; Mon, 3 Mar 2003 16:42:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h23LfTp28687 for ; Mon, 3 Mar 2003 16:41:29 -0500 Received: from palrel10.hp.com (palrel10.hp.com [156.153.255.245]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA13548 for ; Mon, 3 Mar 2003 16:30:48 -0500 (EST) Received: from wera.hpl.hp.com (wera.hpl.hp.com [15.9.120.80]) by palrel10.hp.com (Postfix) with ESMTP id 609B81C0175A for ; Mon, 3 Mar 2003 13:32:50 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by wera.hpl.hp.com (8.12.3/8.12.2) with SMTP id h23LWnSj019267 for rddp@ietf.org; Mon, 3 Mar 2003 13:32:50 -0800 (PST) From: Jeffrey Mogul Message-Id: <200303032132.h23LWnSj019267@wera.hpl.hp.com> X-Authentication-Warning: wera.hpl.hp.com: localhost [127.0.0.1] didn't use HELO protocol To: rddp@ietf.org X-Organization: Hewlett-Packard Labs, Large Scale Systems group Date: Mon, 03 Mar 2003 13:32:49 -0800 X-Mts: smtp Subject: [rddp] Reminder: NICELI Workshop on Network-I/O Convergence (deadline Mar 17) Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , [I'd like to remind people in the RDDP working group about this workshop; the deadline (no extensions!) for submissions is in two weeks.] Inspired by the vigorous technical discussion on this mailing list, and on other related lists, Allyn Romanow and I are organizing a workshop directly concerned with RDDP, RDMA, and related technologies. We encourage anyone with something interesting to consider submitting a paper or position statement. Here's a short blurb: Call For Papers: Workshop on Network-I/O Convergence: Experience, Lessons, Implications (NICELI) in conjunction with SIGCOMM 2003, Karlsruhe, Germany, August, 2003 The performance and commodity price advantages of modern LANs have created a convergence of networks and I/O. This convergence promises both price efficiencies and true interoperability, for storage and for cluster interconnect. The NICELI workshop provides a forum for researchers and practitioners to discuss the merits, drawbacks, applications, and practical implications of protocol and implementation designs. Approaches based on Internet protocols are of particular interest. We invite submissions of research results, protocol design rationales, significant implementation experience, and architectural papers related to the convergence of networks and interconnect. Both technical papers and position statements are welcome. Deadline: Mar. 17 2003 Program co-chairs: Allyn Romanow (Cisco) and Jeff Mogul (HP Labs) See for more details. -Jeff Mogul & Allyn Romanow _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Mar 4 08:18:22 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA18710 for ; Tue, 4 Mar 2003 08:18:22 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h24DSof03229 for rddp-archive@odin.ietf.org; Tue, 4 Mar 2003 08:28:50 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h24DSop03226 for ; Tue, 4 Mar 2003 08:28:50 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA18662 for ; Tue, 4 Mar 2003 08:17:49 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h24DS9p03132; Tue, 4 Mar 2003 08:28:09 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h24DRUp03100 for ; Tue, 4 Mar 2003 08:27:30 -0500 Received: from stewart.chicago.il.us (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA18566 for ; Tue, 4 Mar 2003 08:16:25 -0500 (EST) Received: from stewart.chicago.il.us (stewart.chicago.il.us [127.0.0.1]) by stewart.chicago.il.us (8.12.6/8.12.3) with ESMTP id h24DHjNS012380; Tue, 4 Mar 2003 07:17:48 -0600 (CST) (envelope-from randall@stewart.chicago.il.us) Message-ID: <3E64A779.6050203@stewart.chicago.il.us> Date: Tue, 04 Mar 2003 07:17:45 -0600 From: Randall Stewart User-Agent: Mozilla/5.0 (X11; U; FreeBSD i386; en-US; rv:1.1) Gecko/20021005 X-Accept-Language: en-us, en MIME-Version: 1.0 To: Caitlin Bestler CC: "Williams, Jim" , "'rddp@ietf.org'" Subject: Re: [rddp] SCTP mapping - Multiple Streams References: Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Caitlin: A minor point added to your excellent comments... Caitlin Bestler wrote: > On 3/3/03, Williams, Jim wrote: > > >>>>I am guessing that most implementations will chose to >>>>support one stream per association because it is simpler >>>>and I don't see a performance down side. However since >>>>number of streams per association is negotiable, it is >>>>fine to allow more. >>> >>>On the contrary, there are major advantages to sharing >>>the same association across multiple DDP streams >>>consolidates acking, congestion control and setup. >> >>The down side for the implementation is that two contexts >>(stream and association) must be accessed for each >>packet sent and received rather than one combined context. >>This is a factor of two "cost" increase in one of the >>more expensive aspects of protocol acceleration. As >>an implementer, I would not lightly incur this. >>I don't see the advantages as particularly compelling. >> > > For an established association[1], the association is easily > found with the verification tag. From there the > source/destination addresses should be verified to > filter out spoofed packets. > > That is true whether you have a "consolidated" context > or not. > >>From there you need to perform per-stream processing. > This is a simple lookup within the context of the already > found association. This is the *only* step that having a > one-association / one-stream pairing would save. > > Meanwhile it would avoid savings in association setup > (a four-way handshake), Acks (the overhead of which could > be shared with multiple streams) and congestion control > (each stream would have to do its own slow start). > > [1] An established association is where the performance > issues are most important. Matching an incoming SCTP packet > during association setup and teardown is more complex, as > is currently being discussed in the sctp-implementors list. > > Another savings would be the amount of state.. A SCTP association (or TCP connection for that matter) is a fairly heavy weight thing.. it takes memory to hold all of that state blob that makes up the association/connection. On the other hand, each stream usually takes around 8 - 16 bytes (depending on the implementation.. and I bet you could get it real low if you had the unordered optimization you mention previously)... As such this is much smaller than having N * sizeof(tcb) connections... Quite a signifigant savings... And I think the talk on the implementors guide as come to a conclusion... as Randy Bush as been noted to say "don't feed the trolls"... and I think we have done that for long enough :> > > >>>An integrated RDDP/SCTP implementation SHOULD NOT >>>presume there is only one stream per association unless >>>it has very strong reasons to believe that there will >>>only be a single stream between the two endpoints. >> >>My understanding is that one end of an SCTP association >>may specify one stream per association and the other >>end has no choice but to comply as the number is negotiated >>by the minimum of the two. So there is no problem in >>making such an presumption. The implementation is >>simpler if separate associations are used for each >>stream between endpoints. >> > > Yes that would be allowed. But that presumes that the > association creator knows that there will be no other > streams. How would it know this? > > The key issue is who is the owner of the association? > The ULP or the RDDP stack? If there are multiple clients > that want to access the same remote service, SHOULD they > use the same association? > > As I read the SCTP design, the applicability statement > and the sockets API, I believe there is an intent that > an association should be shared by multiple streams. > One of the major comparison points versus TCP cited is > that applications such as HTTP force applications to create > redundant parallel connections, which is unfair and > unproductive use of network resources. > >>Whether an implementation chooses to do one stream >>per association or multiple streams per associations >>is a design choice that does not need to be mandated >>either way by the mapping standard, and is not mandated >>by the proposal. >> > > I agree that mandating is probably excessive, but I believe > it ought to remain at the SHOULD level. One of the intents > of SCTP is to consolidate the overhead created by parallel > connections. RDDP's mapping should be consistent with this. > > >>However I would prefer to see the number of streams >>per association layered opaquely below DDP so that >>the DDP layer does not know or care about which of its >>streams share or do not share an association. >>Doing otherwise unnecessarily complicates the layering >>model with no added value. Otherwise you end with >>a situation where moving from RDDP/TCP to RDDP/SCTP >>requires unnecessary application changes. This is not >>a good thing. > > > That is a valid point. But one that can be solved by simply > viewing the stream as an extension of the port number. The > "connection qualifier" is the port number + stream number, > where TCP only supports stream 0. > Which then would combine them into a nice 32 bit number that could be hashed ... something that seems to be would be no more overhead than having multiple associations... R > This approach can be extremely valuable when the client > machine uses multiple DDP streams to the same server. For > example a web server accessing a file server via RDDP/SCTP. > Traffic shaping can be performed for the total workload, > whille each individual HTTP fetch is too short to be > effectively traffic shaped. No per-fetch setup is required. > And each external HTTP connection can use its own DDP stream > to interact with the file server (avoiding the need for > a central queuing of file server requests). > > In any event, I think you are correct that this issue cannot > be properly discussed in the absence of some discussion of > the local interface. I believe this requires a discussion > of how a ULP would create a DDP Stream *without* knowledge > of the underlying LLP. > > I would propose that the aforementioned "connection > qualifier" is one very simple method of doing so. The "LLP > Stream" is identified by the remote IP Address(es), the LLP > protocol (SCTP or TCP), the port and the stream number. > > The stack would create the association if this was the first > DDP stream requiring it. Later, it could be deleted when all > of its streams had been terminated. > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > > -- Randall R. Stewart randall@stewart.chicago.il.us 815-342-5222 (cell phone) _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Mar 4 19:33:20 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA17059 for ; Tue, 4 Mar 2003 19:33:20 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h250i2E28999 for rddp-archive@odin.ietf.org; Tue, 4 Mar 2003 19:44:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h250i1528996 for ; Tue, 4 Mar 2003 19:44:01 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA17055 for ; Tue, 4 Mar 2003 19:32:49 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h250eE528834; Tue, 4 Mar 2003 19:40:14 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h250d2528759 for ; Tue, 4 Mar 2003 19:39:02 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA16962 for ; Tue, 4 Mar 2003 19:27:49 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Tue, 4 Mar 2003 19:29:51 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C9AD@corpmx14.us.dg.com> To: rddp@ietf.org Date: Tue, 4 Mar 2003 19:26:47 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] RDDP SF Agenda Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , The RDDP WG will meet twice in San Francisco, 1700 to 1800 on Tuesday, March 18, and 1530 to 1730 on Thursday, March 20. The Problem Statement and Architecture drafts: - draft-ietf-rddp-problem-statement-01.txt - draft-ietf-rddp-arch-01.txt will be WG Last Called on this list within the next few days (WG Last Call will end after the SF meeting week) and will not appear on the RDDP SF meeting agenda unless significant technical issues arise on the mailing list in the interim. The DDP and RDMAP drafts: - draft-ietf-rddp-ddp-00.txt - draft-ietf-rddp-rdmap-00.txt will have brief time slots to describe what has changed since Atlanta and deal with any open technical issues. The SCTP mapping and generic applicability statement drafts: - draft-stewart-rddp-sctp-02.txt - draft-bestler-rddp-applicability-00.txt are new since Atlanta, and will have meeting time on Tuesday to discuss their contents, deal with any technical issues, and determine next steps. Requests will be made to accept these as official RDDP WG drafts. The bulk of the 2 hour meeting on Thursday will be spent on TCP mapping issues, to which the following three drafts are relevant: - draft-culley-iwarp-mpa-02.txt - draft-williams-iwarp-ift-01.txt - draft-elzur-iwarp-mpa-tcp-analysis-00.txt Both the Culley (MPA) and Williams (IFT) drafts have changed since Atlanta. The Elzur draft provides some rationale for changes to the Culley (MPA) draft - it is for the information of WG participants, but will not be "presented" at the meeting. The goal of the time spent on TCP mapping is to resolve as much of the open issues blocking WG progress in this area as possible. The WG's ability to make progress in this area is likely to be a factor considered in the rechartering of the WG that is scheduled to happen this month. Expect to see an agenda structure that focuses attention on technical issues rather than merits of individual drafts. If there are any other drafts that need meeting time (or other requests), please let me know ASAP. Thanks, --David (RDDP WG chair) ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Mar 5 06:40:58 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA27260 for ; Wed, 5 Mar 2003 06:40:58 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h25Bptr18268 for rddp-archive@odin.ietf.org; Wed, 5 Mar 2003 06:51:55 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25Bpt518265 for ; Wed, 5 Mar 2003 06:51:55 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA27222 for ; Wed, 5 Mar 2003 06:40:26 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25BpF518151; Wed, 5 Mar 2003 06:51:15 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25Bi3517478 for ; Wed, 5 Mar 2003 06:44:03 -0500 Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA26641; Wed, 5 Mar 2003 06:32:34 -0500 (EST) Message-Id: <200303051132.GAA26641@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: rddp@ietf.org From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Date: Wed, 05 Mar 2003 06:32:34 -0500 Subject: [rddp] I-D ACTION:draft-ietf-rddp-arch-01.txt Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Remote Direct Data Placement Working Group of the IETF. Title : The Architecture of Direct Data Placement (DDP)And Remote Direct Memory Access (RDMA)On Internet Protocols Author(s) : S. Bailey, T. Talpey Filename : draft-ietf-rddp-arch-01.txt Pages : 17 Date : 2003-3-4 This document defines an abstract architecture for Direct Data Placement (DDP) and Remote Direct Memory Access (RDMA) protocols to run on Internet Protocol-suite transports. This architecture does not necessarily reflect the proper way to implement such protocols, but is, rather, a descriptive tool for defining and understanding the protocols. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-rddp-arch-01.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-rddp-arch-01.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-rddp-arch-01.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <2003-3-4174511.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-rddp-arch-01.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-rddp-arch-01.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <2003-3-4174511.I-D@ietf.org> --OtherAccess-- --NextPart-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Mar 5 07:43:11 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA27265 for ; Wed, 5 Mar 2003 06:40:58 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h25Bpt718287 for rddp-archive@odin.ietf.org; Wed, 5 Mar 2003 06:51:55 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25Bpt518281 for ; Wed, 5 Mar 2003 06:51:55 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA27223 for ; Wed, 5 Mar 2003 06:40:26 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25BpE518135; Wed, 5 Mar 2003 06:51:14 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25Bht517452 for ; Wed, 5 Mar 2003 06:43:55 -0500 Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA26611; Wed, 5 Mar 2003 06:32:27 -0500 (EST) Message-Id: <200303051132.GAA26611@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: rddp@ietf.org From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Date: Wed, 05 Mar 2003 06:32:27 -0500 Subject: [rddp] I-D ACTION:draft-ietf-rddp-problem-statement-01.txt Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Remote Direct Data Placement Working Group of the IETF. Title : RDMA over IP Problem Statement Author(s) : A. Romanow, J. Mogul, T. Talpey, S. Bailey Filename : draft-ietf-rddp-problem-statement-01.txt Pages : 18 Date : 2003-3-4 This draft addresses an IP-based solution to the problem of high system costs due to network I/O copying in end-hosts at high speeds. The problem is due to the high cost of memory bandwidth, and it can be substantially improved using 'copy avoidance.' The high overhead has prevented TCP/IP from being used as an interconnection network. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-rddp-problem-statement-01.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-rddp-problem-statement-01.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-rddp-problem-statement-01.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <2003-3-4174502.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-rddp-problem-statement-01.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-rddp-problem-statement-01.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <2003-3-4174502.I-D@ietf.org> --OtherAccess-- --NextPart-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Mar 5 10:00:52 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA02518 for ; Wed, 5 Mar 2003 10:00:52 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h25FBqA00782 for rddp-archive@odin.ietf.org; Wed, 5 Mar 2003 10:11:52 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25FBq500779 for ; Wed, 5 Mar 2003 10:11:52 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA02404 for ; Wed, 5 Mar 2003 10:00:21 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25FB9500737; Wed, 5 Mar 2003 10:11:09 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25FA2500668 for ; Wed, 5 Mar 2003 10:10:02 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA02328 for ; Wed, 5 Mar 2003 09:58:30 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.12.8/8.12.8) with ESMTP id h25F0SBf004824; Wed, 5 Mar 2003 07:00:28 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Wed, 5 Mar 2003 10:00:28 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B9012@xbl.ma.emulex.com> From: "Williams, Jim" To: "'Randall Stewart'" , Caitlin Bestler Cc: "'rddp@ietf.org'" Subject: RE: [rddp] SCTP mapping - Multiple Streams Date: Wed, 5 Mar 2003 10:00:18 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Randall Stewart wrote: > Caitlin Bestler wrote: > > From there you need to perform per-stream processing. > > This is a simple lookup within the context of the already > > found association. This is the *only* step that having a > > one-association / one-stream pairing would save. > > > > Meanwhile it would avoid savings in association setup > > (a four-way handshake), Acks (the overhead of which could > > be shared with multiple streams) and congestion control > > (each stream would have to do its own slow start). > > > > [1] An established association is where the performance > > issues are most important. Matching an incoming SCTP packet > > during association setup and teardown is more complex, as > > is currently being discussed in the sctp-implementors list. > > > > Another savings would be the amount of state.. A SCTP > association (or TCP connection for that matter) is a > fairly heavy weight thing.. it takes memory to hold > all of that state blob that makes up the association/connection. > On the other hand, each stream usually takes around 8 - 16 > bytes (depending on the implementation.. and I bet you > could get it real low if you had the unordered optimization > you mention previously)... As such this is much smaller > than having N * sizeof(tcb) connections... Quite a > signifigant savings... I think you present some of the design tradeoffs and there's certainly merit in what you say. I might tend to evaluate the tradeoffs a bit differently, but fortunately we don't need to resolve that as implementations may choose different paths. Reasonable people can differ on this (and unreasonable people like me can really differ :). I would suggest the following thought experiment as an interesting diversion if nothing else. Assume that the cost of de-referencing any pointer is (1000 + N), where N is the size in bytes of the object which is pointed to. If the object is variable size, then N is the maximum size the object is allowed to have, not the actual size. Once the cost of resolving the pointer is paid, computation using the contents of the object, and modification of the contents of the object are free. Subject to the above, think about what the minimum cost implementation of SCTP would look like. The above is certainly a gross oversimplification of the behavior of hardware accelerated protocol processors, but gives some flavor of the differences from traditional software implementations. It also might give some clue as to why I have raised the issues that I have. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Mar 5 10:26:52 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA04916 for ; Wed, 5 Mar 2003 10:26:52 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h25FbrS03876 for rddp-archive@odin.ietf.org; Wed, 5 Mar 2003 10:37:53 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25Fbq503873 for ; Wed, 5 Mar 2003 10:37:52 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA04820 for ; Wed, 5 Mar 2003 10:26:18 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25Fb2503103; Wed, 5 Mar 2003 10:37:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25Fab503019 for ; Wed, 5 Mar 2003 10:36:37 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA04631 for ; Wed, 5 Mar 2003 10:25:05 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Wed, 5 Mar 2003 10:27:07 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C9B1@corpmx14.us.dg.com> To: rddp@ietf.org Date: Wed, 5 Mar 2003 10:24:03 -0500 Importance: high X-Priority: 1 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] WG Last Call: Problem Statement and Architecture Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This message announces an informal Working Group Last Call on the RDDP problem statement and architecture drafts that are intended to become informational RFCs: - RDMA over IP Problem Statement, draft-ietf-rddp-problem-statement-01.txt - The Architecture of Direct Data Placement (DDP) And Remote Direct Memory Access (RDMA) On Internet Protocols, draft-ietf-rddp-arch-01.txt Due to the timing of the IETF meetings in San Francisco, this WG Last Call will run slightly longer than typical - it will end at 9am Eastern Time on Monday, March 31st, giving everyone a full week after the IETF meetings to review and comment on these drafts. Editorial comments should be sent directly to the draft editor: Thomas Talpey [Thomas.Talpey@netapp.com]. Technical comments and requests for changes must be sent to the RDDP mailing list [rddp@ietf.org] for discussion. Thanks, --David [RDDP WG chair] ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Mar 5 11:57:42 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA12656 for ; Wed, 5 Mar 2003 11:57:42 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h25H8jo15518 for rddp-archive@odin.ietf.org; Wed, 5 Mar 2003 12:08:45 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25H8j515515 for ; Wed, 5 Mar 2003 12:08:45 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA12593 for ; Wed, 5 Mar 2003 11:57:10 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25H85515427; Wed, 5 Mar 2003 12:08:05 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25H0P513289 for ; Wed, 5 Mar 2003 12:00:25 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA11790 for ; Wed, 5 Mar 2003 11:48:51 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 07B072449B8; Wed, 5 Mar 2003 11:50:54 -0500 (EST) Date: Wed, 5 Mar 2003 10:52:24 -0600 From: Caitlin Bestler Subject: RE: [rddp] SCTP mapping - Multiple Streams To: "'rddp@ietf.org'" Cc: "'Randall Stewart'" , "Williams, Jim" X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B9012@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 3/5/03, Williams, Jim wrote: >Randall Stewart wrote: >> Caitlin Bestler wrote: >> > From there you need to perform per-stream processing. >> > This is a simple lookup within the context of the >> > already found association. This is the *only* step >> > that having a one-association / one-stream pairing >> > would save. >> > >> > Meanwhile it would avoid savings in association setup >> > (a four-way handshake), Acks (the overhead of which >> > could be shared with multiple streams) and congestion >> > control (each stream would have to do its own slow >> > start). >> > >> > [1] An established association is where the >> > performance issues are most important. Matching an >> > incoming SCTP packet during association setup and >> > teardown is more complex, as is currently being >> > discussed in the sctp-implementors list. >> > >> >> Another savings would be the amount of state.. A SCTP >> association (or TCP connection for that matter) is a >> fairly heavy weight thing.. it takes memory to hold all >> of that state blob that makes up the >> association/connection. >> On the other hand, each stream usually takes around 8 - >> 16 bytes (depending on the implementation.. and I bet >> you could get it real low if you had the unordered >> optimization you mention previously)... As such this is >> much smaller than having N * sizeof(tcb) connections... >> Quite a signifigant savings... > >I think you present some of the design tradeoffs and >there's certainly merit in what you say. I might tend to >evaluate the tradeoffs a bit differently, but fortunately >we don't need to resolve that as implementations may >choose different paths. Reasonable people can differ on >this (and unreasonable people like me can really differ :). > >I would suggest the following thought experiment as an >interesting diversion if nothing else. > > Assume that the cost of de-referencing any pointer > is (1000 + N), where N is the size in bytes of the > object which is pointed to. If the object is variable > size, then N is the maximum size the object is > allowed to have, not the actual size. Once the cost > of resolving the pointer is paid, computation using > the contents of the object, and modification of > the contents of the object are free. > > Subject to the above, think about what the minimum > cost implementation of SCTP would look like. > >The above is certainly a gross oversimplification >of the behavior of hardware accelerated protocol >processors, but gives some flavor of the differences >from traditional software implementations. It also >might give some clue as to why I have raised the >issues that I have. OK, let me be more explicit about the degreee of optimization that is possible. The SCTP overhead per stream for a DDP-apapted association can in fact be as low as zero. DDP produces only Data Chunks which are: -- complete, both the B and E bits are set. All fragmentation is either performed at the DDP layer or at the IP layer. Each SCTP Data Chunk represents a complete DDP Segment. -- unordered Therefore the SCTP layer can always deliver the Data Chunk to the LLP immediately. There is no per-stream processing or reordering required. The Stream ID is essentially nothing more than pass-through ULP data. (With the execption that the sender MAY perform some form of fairness allocation across streams). The 8 to 16 bytes that Randall referred to are only needed to support ordered delivery. Any implementation that defers creation of this data until a stream is actually used will have *zero* SCTP overhead from the mere existence of a stream. More importantly, whether or not the data structure to support ordering for stream X exists, there is no need to consult it when placing unordered data for stream X. The bottom line: for placement of unordered data chunks the Stream ID has no signifigance to the SCTP layer. It is merely passed through to the DDP receiver. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Mar 5 18:33:06 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA06247 for ; Wed, 5 Mar 2003 18:33:06 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h25Nhpb02462 for rddp-archive@odin.ietf.org; Wed, 5 Mar 2003 18:43:51 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25NhpO02459 for ; Wed, 5 Mar 2003 18:43:51 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA06207 for ; Wed, 5 Mar 2003 18:32:34 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25NhBO02422; Wed, 5 Mar 2003 18:43:11 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h25NgGO02400 for ; Wed, 5 Mar 2003 18:42:16 -0500 Received: from palrel10.hp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA06129 for ; Wed, 5 Mar 2003 18:31:00 -0500 (EST) Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel10.hp.com (Postfix) with ESMTP id D5B951C00C13 for ; Wed, 5 Mar 2003 15:33:03 -0800 (PST) Received: from mk731915.cup.hp.com ([15.113.161.2]) by esmail.cup.hp.com (8.11.1/8.8.6) with ESMTP id h25NUCL00191 for ; Wed, 5 Mar 2003 15:30:12 -0800 (PST) Message-Id: <5.1.0.14.2.20030305153154.037a4c28@esmail.cup.hp.com> X-Sender: krause@esmail.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Wed, 05 Mar 2003 15:32:28 -0800 To: rddp@ietf.org From: Michael Krause Subject: RE: [rddp] Marker Interval In-Reply-To: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 11:37 AM 2/24/2003 -0800, Larsen, Roy K wrote: > > -----Original Message----- > > From: Michael Krause [mailto:krause@cup.hp.com] > > > > At 09:42 AM 2/24/2003 -0800, Somesh Gupta wrote: > > >The discussion is related to existing software stacks and > > >interfaces. For storage protocols (as is for iSCSI), this > > >forms the first wave of client side implementations. > > > > Again this is a function of implementation and a well-layered > > implementation can be delivered without requiring any network stack > > modifications if desired. Same applies to software based > > iSCSI which has > > been demonstrated by multiple vendors without network stack > > modifications. Some of these implementations now deliver sufficient > > performance that current customers using them are not raising > > any major > > objections. Given this can be done with iSCSI and markers > > are clearly less > > complex to implement / support in software than iSCSI, one > > might argue that > > the complexity / cost issues are not as high as some assert. > >Mike, > >Marker use in iSCSI is a negotiated option. Do you know if these successful >software iSCSI implementations are running with markers enabled? We have implemented it (was not that complex for our initiator) but have not tested it with a target as yet. Mike _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Mar 10 14:35:57 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA09801 for ; Mon, 10 Mar 2003 14:35:57 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h2AJn3Y08960 for rddp-archive@odin.ietf.org; Mon, 10 Mar 2003 14:49:03 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2AJn3O08957 for ; Mon, 10 Mar 2003 14:49:03 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA09776 for ; Mon, 10 Mar 2003 14:35:26 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2AJjEO08852; Mon, 10 Mar 2003 14:45:14 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2AJiSO08816 for ; Mon, 10 Mar 2003 14:44:28 -0500 Received: from mxic2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA09519; Mon, 10 Mar 2003 14:30:52 -0500 (EST) From: Black_David@emc.com Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Mon, 10 Mar 2003 14:32:06 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C9F2@corpmx14.us.dg.com> To: agenda@ietf.org Cc: rddp@ietf.org Date: Mon, 10 Mar 2003 14:29:36 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] Remote Direct Data Placement (rddp) San Francisco Agenda Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , San Francisco IETF Meeting March 16-21, 2003 --------------------------- Remote Direct Data Placement WG (rddp) CHAIR: David L. Black The RDDP WG works on Remote Direct Data Placement and Remote Direct Memory Access functionality. For more information, please refer to the rddp charter at: http://www.ietf.org/html.charters/rddp-charter.html NOTE: The following two drafts are in WG Last Call, end at 9am Eastern Time on Monday, March 31st. Send comments to the mailing list, rddp@ietf.org: - RDMA over IP Problem Statement, draft-ietf-rddp-problem-statement-01.txt - The Architecture of Direct Data Placement (DDP) And Remote Direct Memory Access (RDMA) On Internet Protocols, draft-ietf-rddp-arch-01.txt AGENDA: Tuesday, March 18, 1700-1800 ---------------------------- - Agenda Bashing and Administrivia (10 min) - Blue Sheets - NOTE WELL - Status of drafts - SCTP Mapping (20 min) (draft-stewart-rddp-sctp-02.txt) - Applicability Statement (15 min) (draft-bestler-rddp-applicability-00.txt) - TCP Mapping Requirements (15 min) [tentative] Discussion of requirements on the WG's TCP Mapping, including framing work. WARNING: TCP Mapping Requirements may get moved to between RDMAP and TCP mapping on Thursday, in which case its current time slot will be used for DDP. Thursday, March 20, 1530-1730 ----------------------------- - Agenda Re-Bashing and Administrivia (5 min) - DDP (15 min) (draft-ietf-rddp-ddp-00.txt) - RDMAP (15 min) (draft-ietf-rddp-rdmap-00.txt) - TCP mapping (85 min) (draft-culley-iwarp-mpa-02.txt) (draft-williams-iwarp-ift-01.txt) (draft-elzur-iwarp-mpa-tcp-analysis-00.txt) Both the Culley (MPA) and Williams (IFT) drafts have changed since Atlanta. The Elzur draft provides some rationale for changes to the Culley (MPA) draft - it is for the information of WG participants, but will not be "presented" at the meeting. The goal of this time is to resolve as much of the open issues blocking WG progress in this area as possible. The WG's ability to make progress in this area is likely to be a factor considered in the rechartering of the WG that is scheduled to happen this month. Expect to see this time structured to focus attention on technical issues rather than a comparison of the drafts. NOTE: Attendees should also read the following draft as background for the meeting, although no agenda time is allocated to it: (1) draft-ietf-rddp-rdma-concerns-00.txt Describes concerns that rddp work needs to address. ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Mar 14 09:36:09 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA26284 for ; Fri, 14 Mar 2003 09:36:09 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h2EEp6K14856 for rddp-archive@odin.ietf.org; Fri, 14 Mar 2003 09:51:06 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2EEp6O14853 for ; Fri, 14 Mar 2003 09:51:06 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA26259 for ; Fri, 14 Mar 2003 09:35:37 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2EEoLO14808; Fri, 14 Mar 2003 09:50:21 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2EEnxO14738 for ; Fri, 14 Mar 2003 09:49:59 -0500 Received: from mx01.netapp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA26232 for ; Fri, 14 Mar 2003 09:34:31 -0500 (EST) Received: from frejya.corp.netapp.com (frejya [10.10.20.91]) by mx01.netapp.com (8.12.8/8.12.8/NTAP-1.4) with ESMTP id h2EEafvt018281 for ; Fri, 14 Mar 2003 06:36:41 -0800 (PST) Received: from tmt.netapp.com (tmt [10.97.1.30]) by frejya.corp.netapp.com (8.12.8/8.12.8/NTAP-1.4) with ESMTP id h2EEaexv003073 for ; Fri, 14 Mar 2003 06:36:40 -0800 (PST) Message-Id: <5.2.0.9.2.20030314093345.04203998@silver.nane.netapp.com> Date: Fri, 14 Mar 2003 09:36:10 -0500 To: rddp@ietf.org From: "Talpey, Thomas" Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: [rddp] RDDP session multicast Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Looks like the longer of the two RDDP WG sessions will be multicast: THURSDAY 0900-1130 1300-1500 1530-1730 1930-2200 (UTC) 1700-1930 2100-2300 2330-0130 0330-0600 =========+==========+==========+==========+==========+ CHAN 1 | avt | tsvwg | rddp | plenary | =========+==========+==========+==========+==========+ CHAN 2 | ipv6 | opsarea | saag | plenary | =========+==========+==========+==========+==========+ FYI. Tom. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Mar 27 15:42:31 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA18427 for ; Thu, 27 Mar 2003 15:42:31 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h2RL3wU19972 for rddp-archive@odin.ietf.org; Thu, 27 Mar 2003 16:03:58 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2RL3wO19969 for ; Thu, 27 Mar 2003 16:03:58 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA18421 for ; Thu, 27 Mar 2003 15:42:00 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2RL32O19927; Thu, 27 Mar 2003 16:03:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2RL1ZO19821 for ; Thu, 27 Mar 2003 16:01:35 -0500 Received: from mx01.netapp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA18371 for ; Thu, 27 Mar 2003 15:39:36 -0500 (EST) Received: from frejya.corp.netapp.com (frejya [10.10.20.91]) by mx01.netapp.com (8.12.8/8.12.8/NTAP-1.4) with ESMTP id h2RKfvqU020901 for ; Thu, 27 Mar 2003 12:41:58 -0800 (PST) Received: from tmt.netapp.com (tmt [10.97.1.30]) by frejya.corp.netapp.com (8.12.8/8.12.8/NTAP-1.4) with ESMTP id h2RKfuKo011822 for ; Thu, 27 Mar 2003 12:41:57 -0800 (PST) Message-Id: <5.2.0.9.2.20030327152808.00af4208@silver.nane.netapp.com> Date: Thu, 27 Mar 2003 15:41:01 -0500 To: rddp@ietf.org From: "Talpey, Thomas" Subject: Re: [rddp] WG Last Call: Problem Statement and Architecture In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C9B1@corpmx14.us.dg.co m> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , A reminder, and notice that there is one issue in the Architecture document which will be addressed in the final version. The issue is around the change made in the current document to add a ddp_post_recv() function. From a pure architecture standpoint this has been deemed unnecessary, and the previous form of ddp_recv() will be restored, along with some new explanatory text. In effect, this returns to the ddp_recv() formal function returning a message_t, rather than a necessarily pre-registered recv_message_t. The difference is subtle but important. It is not necessary to make the requirement of preregistration for receives in the architecture, therefore the simplicity argument prevails. Any comments or concerns on the reversion to the prior ddp_recv() behavior should be sent to the list as soon as possible. There are currently no additional outstanding comments or changes on either document. Thanks, Tom. At 10:24 AM 3/5/2003, Black_David@emc.com wrote: >This message announces an informal Working Group >Last Call on the RDDP problem statement and architecture >drafts that are intended to become informational RFCs: > >- RDMA over IP Problem Statement, > draft-ietf-rddp-problem-statement-01.txt >- The Architecture of Direct Data Placement (DDP) And > Remote Direct Memory Access (RDMA) On Internet > Protocols, draft-ietf-rddp-arch-01.txt > >Due to the timing of the IETF meetings in San Francisco, >this WG Last Call will run slightly longer than typical - >it will end at 9am Eastern Time on Monday, March 31st, >giving everyone a full week after the IETF meetings >to review and comment on these drafts. > >Editorial comments should be sent directly to the >draft editor: Thomas Talpey [Thomas.Talpey@netapp.com]. >Technical comments and requests for changes must be >sent to the RDDP mailing list [rddp@ietf.org] for >discussion. > >Thanks, >--David [RDDP WG chair] > >---------------------------------------------------- >David L. Black, Senior Technologist >EMC Corporation, 176 South St., Hopkinton, MA 01748 >+1 (508) 293-7953 FAX: +1 (508) 293-7786 >black_david@emc.com Mobile: +1 (978) 394-7754 >---------------------------------------------------- >_______________________________________________ >rddp mailing list >rddp@ietf.org >https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Mar 27 16:02:57 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA19007 for ; Thu, 27 Mar 2003 16:02:57 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h2RLOO721750 for rddp-archive@odin.ietf.org; Thu, 27 Mar 2003 16:24:24 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2RLOOO21747 for ; Thu, 27 Mar 2003 16:24:24 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA18998 for ; Thu, 27 Mar 2003 16:02:26 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2RLNZO21678; Thu, 27 Mar 2003 16:23:35 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2RLMoO21649 for ; Thu, 27 Mar 2003 16:22:50 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA18949 for ; Thu, 27 Mar 2003 16:00:52 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 1F11424563D; Thu, 27 Mar 2003 16:03:22 -0500 (EST) Date: Thu, 27 Mar 2003 15:05:26 -0600 From: Caitlin Bestler Subject: Re: [rddp] WG Last Call: Problem Statement and Architecture To: "Talpey, Thomas" Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <5.2.0.9.2.20030327152808.00af4208@silver.nane.netapp.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 3/27/03, Talpey, Thomas wrote: >A reminder, and notice that there is one issue in the >Architecture document which will be addressed in the >final version. > >The issue is around the change made in the current >document to add a ddp_post_recv() function. From a >pure architecture standpoint this has been deemed >unnecessary, and the previous form of ddp_recv() >will be restored, along with some new explanatory >text. In effect, this returns to the ddp_recv() formal >function returning a message_t, rather than a >necessarily pre-registered recv_message_t. > >The difference is subtle but important. It is not >necessary to make the requirement of preregistration >for receives in the architecture, therefore the simplicity >argument prevails. > >Any comments or concerns on the reversion to the >prior ddp_recv() behavior should be sent to the list as >soon as possible. There are currently no additional >outstanding comments or changes on either document. > The iWARP suite relies upon the ULP to provide flow control. It has no end-to-end flow control of its own. Requiring receive buffers to be pre-registered, or even pre-allocated, is a serious issue. It almost always makes sense. However it is definitely not required for interoperability, and there are some applications where it can be shown that use of system buffering for untagged messages would be superior to the traditional RDMA local interface (as inherited from VIA and IB). What I would suggest is that the requirement simply be that the ULP must *authorize* each reception. The local interface MUST allow the ULP to supply the buffer, and MAY require the ULP to do so. The ULP MAY be allowed to pre-post a receive authorization without supplying a pre-committed buffer. This approach would keep the pre-posting of receive operations, without requiring that a pre-registered buffer be supplied. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Mar 27 17:08:57 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA21665 for ; Thu, 27 Mar 2003 17:08:57 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h2RMUO027103 for rddp-archive@odin.ietf.org; Thu, 27 Mar 2003 17:30:24 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2RMUOO27100 for ; Thu, 27 Mar 2003 17:30:24 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA21659 for ; Thu, 27 Mar 2003 17:08:26 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2RMTZO27068; Thu, 27 Mar 2003 17:29:35 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h2RMSeO27016 for ; Thu, 27 Mar 2003 17:28:40 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA21637 for ; Thu, 27 Mar 2003 17:06:40 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 460872449EB for ; Thu, 27 Mar 2003 17:09:10 -0500 (EST) Date: Thu, 27 Mar 2003 16:11:14 -0600 From: Caitlin Bestler To: rddp@ietf.org X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] Pre-posting/authorizing receive operations Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit I'd like to expand a bit on the issue of pre-posting, or at least pre-authorizing of receives. I agree that requiring the buffers to be registered and pre-allocated is incorrect. However, I believe that it is vital that the receiving ULP pre-authorize each untagged message and that excess untagged messages will terminate the DDP Stream rather than consuming system resources and placing *other* DDP Streams at risk. With both the MPA/TCP and SCTP mappings, the transport/DDP interface received DDP Segments can be pushed to the DDP layer. The DDP layer can be expected to place the received data immediately, i.e. the LLP has no further need to hold the buffer after the "push" call completes. Without pre-posting of receive buffers the receiving DDP would be obligated to buffer untagged data until the ULP requested it. This is not something that a DDP implementation should be required to do. That is my justification for saying that a DDP implementation MAY require the ULP to pre-post a buffer for each receive operation. But even if a DDP implementation is willing to use a common buffer pool, it should not allow that pool to be drained by a single remote peer. Requiring the local ULP to at least pre-authorize the receive operation guarantess that there is ULP level flow control on untagged messages. If the local peer attempts to authorize too many receive operations without supplying buffers, the DDP implementation can refuse. This may require the DDP Stream to be aborted -- but the impact is limited to the DDP Stream causing the problem. Without pre-authorization of receives, the DDP Stream that comes up empty on system buffering may not be the guilty party that drained the pool. The reason for not requiring the buffer to be pre-registered is simple. The ULP may have no need to expose that memory for any longer than the single receive operation. The DDP layer already has all of the information required to "register" any buffer for the duration of the read. Any requirement that the buffer be pre-registered would be based upon the local implementations cost of dynamically reqistering the memory. While this may be a valid concern in designing a local interface, it is not relevant for specifying the portions of the local interface that MUST be present to ensure interoperability. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp