From mailnull@www1.ietf.org Tue Feb 4 11:31:36 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA27686 for ; Tue, 4 Feb 2003 11:31:36 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h14Gb8706796 for rddp-archive@odin.ietf.org; Tue, 4 Feb 2003 11:37:08 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h14Gb8J06785 for ; Tue, 4 Feb 2003 11:37:08 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA27674 for ; Tue, 4 Feb 2003 11:31:04 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h14GaNJ06607; Tue, 4 Feb 2003 11:36:23 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h14GZIJ06533 for ; Tue, 4 Feb 2003 11:35:18 -0500 Received: from e34.co.us.ibm.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA27566 for ; Tue, 4 Feb 2003 11:29:13 -0500 (EST) Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e34.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id h14GWoIA040522 for ; Tue, 4 Feb 2003 11:32:50 -0500 Received: from d03nm803.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.3/NCO/VER6.5) with ESMTP id h14GVJe0035286 for ; Tue, 4 Feb 2003 09:31:21 -0700 To: rddp@ietf.org X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: Renato Recio Date: Tue, 4 Feb 2003 10:31:12 -0600 X-MIMETrack: Serialize by Router on D03NM803/03/M/IBM(Release 6.0 [IBM]|December 16, 2002) at 02/04/2003 09:31:21 MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Subject: [rddp] ULP error codes Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Folks, The RDMA specification (-01) does not expose the terminate message to the ULP, because ULPs have their own mechanism for communicating ULP level errors between the two ends of a ULP Stream. The layer performing the RDMAP layer is not a ULP, but will likely have additional, local, functionality beyond the function performed by the RDMAP. For example, if the RDMAP (and lower) layer(s) are implemented on an RNIC, the RNIC may have additional error codes that are not captured by the RDMAP layer error codes currently in the specification, such as an error associated with an invalid state that is internal to the RNIC. These additional error codes would convey diagnostic/debug information that would be useful at both ends of the RDMA Stream being terminated. I would like to propose that we use value 1111b of the terminate layer field for RNIC related error types and codes. I would also like to propose that we state the error type and error code fields are reserved for use by the RNIC when the layer is 1111b. Thanks, Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Feb 5 16:02:06 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA10626 for ; Wed, 5 Feb 2003 16:02:06 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h15L8D223491 for rddp-archive@odin.ietf.org; Wed, 5 Feb 2003 16:08:13 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h15L8CJ23488 for ; Wed, 5 Feb 2003 16:08:12 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA10608 for ; Wed, 5 Feb 2003 16:01:34 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h15L7SJ23378; Wed, 5 Feb 2003 16:07:28 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h15L6JJ22762 for ; Wed, 5 Feb 2003 16:06:19 -0500 Received: from ztxmail05.ztx.compaq.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA10554 for ; Wed, 5 Feb 2003 15:59:41 -0500 (EST) Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125]) by ztxmail05.ztx.compaq.com (Postfix) with ESMTP id 33B172EB6 for ; Wed, 5 Feb 2003 15:03:19 -0600 (CST) Received: from cceexc16.americas.cpqcorp.net ([16.110.250.58]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Wed, 5 Feb 2003 15:03:08 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.6344.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: RE: [rddp] ULP error codes Date: Wed, 5 Feb 2003 15:03:05 -0600 Message-ID: <07DEBFB3DCB18A49B06B6371993586740292FD6E@cceexc16.americas.cpqcorp.net> Thread-Topic: [rddp] ULP error codes Thread-Index: AcLMbIoatEA8p5a3TyyJPI7IMG+JbAA7P2gA From: "Hilland, Jeff" To: X-OriginalArrivalTime: 05 Feb 2003 21:03:08.0833 (UTC) FILETIME=[FC4AA910:01C2CD59] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h15L6JJ22766 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit As you noted, ULPs which are terminating should have their own termination mechanism and so don't need the codes. I don't believe that the RNIC error codes below are necessary either. They certainly would not be uniform across implementations. I believe that the Catastrophic Error code covers these classes of errors. Regards, Jeff Hilland -----Original Message----- From: Renato Recio [mailto:recio@us.ibm.com] Sent: Tuesday, February 04, 2003 10:31 AM To: rddp@ietf.org Subject: [rddp] ULP error codes Folks, The RDMA specification (-01) does not expose the terminate message to the ULP, because ULPs have their own mechanism for communicating ULP level errors between the two ends of a ULP Stream. The layer performing the RDMAP layer is not a ULP, but will likely have additional, local, functionality beyond the function performed by the RDMAP. For example, if the RDMAP (and lower) layer(s) are implemented on an RNIC, the RNIC may have additional error codes that are not captured by the RDMAP layer error codes currently in the specification, such as an error associated with an invalid state that is internal to the RNIC. These additional error codes would convey diagnostic/debug information that would be useful at both ends of the RDMA Stream being terminated. I would like to propose that we use value 1111b of the terminate layer field for RNIC related error types and codes. I would also like to propose that we state the error type and error code fields are reserved for use by the RNIC when the layer is 1111b. Thanks, Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Feb 5 16:43:36 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA12074 for ; Wed, 5 Feb 2003 16:43:36 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h15Lnh225743 for rddp-archive@odin.ietf.org; Wed, 5 Feb 2003 16:49:43 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h15LnhJ25740 for ; Wed, 5 Feb 2003 16:49:43 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA12046 for ; Wed, 5 Feb 2003 16:43:04 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h15Ln2J25694; Wed, 5 Feb 2003 16:49:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h15LmVJ25671 for ; Wed, 5 Feb 2003 16:48:31 -0500 Received: from e34.co.us.ibm.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA12017 for ; Wed, 5 Feb 2003 16:41:52 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.194.23]) by e34.co.us.ibm.com (8.12.2/8.12.2) with ESMTP id h15LjRog050768; Wed, 5 Feb 2003 16:45:27 -0500 Received: from d03nm803.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay02.boulder.ibm.com (8.12.3/NCO/VER6.5) with ESMTP id h15LjQ81168046; Wed, 5 Feb 2003 14:45:26 -0700 Subject: RE: [rddp] ULP error codes To: "Hilland, Jeff" Cc: rddp@ietf.org X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: Renato Recio Date: Wed, 5 Feb 2003 15:45:24 -0600 X-MIMETrack: Serialize by Router on D03NM803/03/M/IBM(Release 6.0 [IBM]|December 16, 2002) at 02/05/2003 14:45:26 MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , I agree with you. I'd suggest covering other RDMA layer errors that can happen as follows: 1. Any errors that don't fall into the category of remote protection and remote operation error, do as you said. That is, surface them as Local Catastrophic Errors. 2. Create one new error code to cover Miscallaneous Remote Protection Errors: Misc for Remote Protection Errors: Layer = 0x RDMA Type = 1x Remote Protection Error Code = FFx Miscallaneous 3. Create one new error code to cover Miscallaneous Remote Operation Errors: Misc for Remote Operation Errors: Layer = 0x RDMA Type = 2x Remote Operation Error Code = FFx Miscallaneous Additionally, I'd suggest adding a new Remote Operation Error error code to cover the case where a Send with Invalidate is received from the remote peer, but the STag cannot be Invalidated because it is in a state where invalidation cannot be performed. Following is the new error code for this case: Layer = 0x RDMA Type = 2x Remote Operation Error Code = 09x STag cannot be Invalidated Thanks, Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 "Hilland, Jeff" com> cc: Sent by: Subject: RE: [rddp] ULP error codes rddp-admin@ietf.o rg 02/05/2003 03:03 PM As you noted, ULPs which are terminating should have their own termination mechanism and so don't need the codes. I don't believe that the RNIC error codes below are necessary either. They certainly would not be uniform across implementations. I believe that the Catastrophic Error code covers these classes of errors. Regards, Jeff Hilland -----Original Message----- From: Renato Recio [mailto:recio@us.ibm.com] Sent: Tuesday, February 04, 2003 10:31 AM To: rddp@ietf.org Subject: [rddp] ULP error codes Folks, The RDMA specification (-01) does not expose the terminate message to the ULP, because ULPs have their own mechanism for communicating ULP level errors between the two ends of a ULP Stream. The layer performing the RDMAP layer is not a ULP, but will likely have additional, local, functionality beyond the function performed by the RDMAP. For example, if the RDMAP (and lower) layer(s) are implemented on an RNIC, the RNIC may have additional error codes that are not captured by the RDMAP layer error codes currently in the specification, such as an error associated with an invalid state that is internal to the RNIC. These additional error codes would convey diagnostic/debug information that would be useful at both ends of the RDMA Stream being terminated. I would like to propose that we use value 1111b of the terminate layer field for RNIC related error types and codes. I would also like to propose that we state the error type and error code fields are reserved for use by the RNIC when the layer is 1111b. Thanks, Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Feb 5 17:30:14 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA13434 for ; Wed, 5 Feb 2003 17:30:14 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h15MaNC02475 for rddp-archive@odin.ietf.org; Wed, 5 Feb 2003 17:36:23 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h15MaNp02472 for ; Wed, 5 Feb 2003 17:36:23 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA13387 for ; Wed, 5 Feb 2003 17:29:42 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h15MZNp02392; Wed, 5 Feb 2003 17:35:23 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h15MYTp02357 for ; Wed, 5 Feb 2003 17:34:29 -0500 Received: from ztxmail05.ztx.compaq.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA13339 for ; Wed, 5 Feb 2003 17:27:49 -0500 (EST) Received: from cceexg13.americas.cpqcorp.net (cceexg13.americas.cpqcorp.net [16.110.250.119]) by ztxmail05.ztx.compaq.com (Postfix) with ESMTP id 2F06D2E71 for ; Wed, 5 Feb 2003 16:31:27 -0600 (CST) Received: from cceexc16.americas.cpqcorp.net ([16.110.250.58]) by cceexg13.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Wed, 5 Feb 2003 16:31:27 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.6344.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: RE: [rddp] ULP error codes Date: Wed, 5 Feb 2003 16:31:16 -0600 Message-ID: <07DEBFB3DCB18A49B06B63719935867403FF5F01@cceexc16.americas.cpqcorp.net> Thread-Topic: [rddp] ULP error codes Thread-Index: AcLNX+ozDReA0tYvSgmezcht9UDdTQABOPbw From: "Hilland, Jeff" To: X-OriginalArrivalTime: 05 Feb 2003 22:31:27.0056 (UTC) FILETIME=[52475500:01C2CD66] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h15MYTp02358 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit Clearly there is a need for the "STag cannot be invalidated" error, since this is a Remote Error covered by the STag to be Invalidated, which is part of the protocol. But this could be a Remote Protection Error if the STag existed but was not valid on the given stream. So we certainly need (0x, 1x, 09x or 0109x). If the STag doesn't exist, then you would have the Remote Operation Error. I like the Miscellaneous category. Regards, Jeff -----Original Message----- From: Renato Recio [mailto:recio@us.ibm.com] Sent: Wednesday, February 05, 2003 3:45 PM To: Hilland, Jeff Cc: rddp@ietf.org Subject: RE: [rddp] ULP error codes I agree with you. I'd suggest covering other RDMA layer errors that can happen as follows: 1. Any errors that don't fall into the category of remote protection and remote operation error, do as you said. That is, surface them as Local Catastrophic Errors. 2. Create one new error code to cover Miscallaneous Remote Protection Errors: Misc for Remote Protection Errors: Layer = 0x RDMA Type = 1x Remote Protection Error Code = FFx Miscallaneous 3. Create one new error code to cover Miscallaneous Remote Operation Errors: Misc for Remote Operation Errors: Layer = 0x RDMA Type = 2x Remote Operation Error Code = FFx Miscallaneous Additionally, I'd suggest adding a new Remote Operation Error error code to cover the case where a Send with Invalidate is received from the remote peer, but the STag cannot be Invalidated because it is in a state where invalidation cannot be performed. Following is the new error code for this case: Layer = 0x RDMA Type = 2x Remote Operation Error Code = 09x STag cannot be Invalidated Thanks, Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 "Hilland, Jeff" com> cc: Sent by: Subject: RE: [rddp] ULP error codes rddp-admin@ietf.o rg 02/05/2003 03:03 PM As you noted, ULPs which are terminating should have their own termination mechanism and so don't need the codes. I don't believe that the RNIC error codes below are necessary either. They certainly would not be uniform across implementations. I believe that the Catastrophic Error code covers these classes of errors. Regards, Jeff Hilland -----Original Message----- From: Renato Recio [mailto:recio@us.ibm.com] Sent: Tuesday, February 04, 2003 10:31 AM To: rddp@ietf.org Subject: [rddp] ULP error codes Folks, The RDMA specification (-01) does not expose the terminate message to the ULP, because ULPs have their own mechanism for communicating ULP level errors between the two ends of a ULP Stream. The layer performing the RDMAP layer is not a ULP, but will likely have additional, local, functionality beyond the function performed by the RDMAP. For example, if the RDMAP (and lower) layer(s) are implemented on an RNIC, the RNIC may have additional error codes that are not captured by the RDMAP layer error codes currently in the specification, such as an error associated with an invalid state that is internal to the RNIC. These additional error codes would convey diagnostic/debug information that would be useful at both ends of the RDMA Stream being terminated. I would like to propose that we use value 1111b of the terminate layer field for RNIC related error types and codes. I would also like to propose that we state the error type and error code fields are reserved for use by the RNIC when the layer is 1111b. Thanks, Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 6 09:52:25 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA19917 for ; Thu, 6 Feb 2003 09:52:25 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h16Ewsm05030 for rddp-archive@odin.ietf.org; Thu, 6 Feb 2003 09:58:54 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h16Ewrp05027 for ; Thu, 6 Feb 2003 09:58:53 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA19897 for ; Thu, 6 Feb 2003 09:51:51 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h16Ew5p04979; Thu, 6 Feb 2003 09:58:05 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h16Evup04962 for ; Thu, 6 Feb 2003 09:57:56 -0500 Received: from e4.ny.us.ibm.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA19894 for ; Thu, 6 Feb 2003 09:50:56 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.194.23]) by e4.ny.us.ibm.com (8.12.7/8.12.2) with ESMTP id h16EsWx7071884; Thu, 6 Feb 2003 09:54:32 -0500 Received: from d03nm803.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay02.boulder.ibm.com (8.12.3/NCO/VER6.5) with ESMTP id h16EqvxM169044; Thu, 6 Feb 2003 07:52:57 -0700 Subject: RE: [rddp] ULP error codes To: "Hilland, Jeff" Cc: rddp@ietf.org X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: Renato Recio Date: Thu, 6 Feb 2003 08:52:56 -0600 X-MIMETrack: Serialize by Router on D03NM803/03/M/IBM(Release 6.0 [IBM]|December 16, 2002) at 02/06/2003 07:52:57 MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Sounds good, I plan to add the following new error codes: Layer = 0x RDMA Type = 1x Remote Protection Error Code = 09x (Name = STag cannot be Invalidated) Code = FFx (Name = Miscallaneous) Layer = 0x RDMA Type = 2x Remote Operation Error Code = 09x (Name = STag cannot be Invalidated) Code = FFx (Name = Miscallaneous) Thanks, Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 "Hilland, Jeff" com> cc: Sent by: Subject: RE: [rddp] ULP error codes rddp-admin@ietf.o rg 02/05/2003 04:31 PM Clearly there is a need for the "STag cannot be invalidated" error, since this is a Remote Error covered by the STag to be Invalidated, which is part of the protocol. But this could be a Remote Protection Error if the STag existed but was not valid on the given stream. So we certainly need (0x, 1x, 09x or 0109x). If the STag doesn't exist, then you would have the Remote Operation Error. I like the Miscellaneous category. Regards, Jeff -----Original Message----- From: Renato Recio [mailto:recio@us.ibm.com] Sent: Wednesday, February 05, 2003 3:45 PM To: Hilland, Jeff Cc: rddp@ietf.org Subject: RE: [rddp] ULP error codes I agree with you. I'd suggest covering other RDMA layer errors that can happen as follows: 1. Any errors that don't fall into the category of remote protection and remote operation error, do as you said. That is, surface them as Local Catastrophic Errors. 2. Create one new error code to cover Miscallaneous Remote Protection Errors: Misc for Remote Protection Errors: Layer = 0x RDMA Type = 1x Remote Protection Error Code = FFx Miscallaneous 3. Create one new error code to cover Miscallaneous Remote Operation Errors: Misc for Remote Operation Errors: Layer = 0x RDMA Type = 2x Remote Operation Error Code = FFx Miscallaneous Additionally, I'd suggest adding a new Remote Operation Error error code to cover the case where a Send with Invalidate is received from the remote peer, but the STag cannot be Invalidated because it is in a state where invalidation cannot be performed. Following is the new error code for this case: Layer = 0x RDMA Type = 2x Remote Operation Error Code = 09x STag cannot be Invalidated Thanks, Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 "Hilland, Jeff" com> cc: Sent by: Subject: RE: [rddp] ULP error codes rddp-admin@ietf.o rg 02/05/2003 03:03 PM As you noted, ULPs which are terminating should have their own termination mechanism and so don't need the codes. I don't believe that the RNIC error codes below are necessary either. They certainly would not be uniform across implementations. I believe that the Catastrophic Error code covers these classes of errors. Regards, Jeff Hilland -----Original Message----- From: Renato Recio [mailto:recio@us.ibm.com] Sent: Tuesday, February 04, 2003 10:31 AM To: rddp@ietf.org Subject: [rddp] ULP error codes Folks, The RDMA specification (-01) does not expose the terminate message to the ULP, because ULPs have their own mechanism for communicating ULP level errors between the two ends of a ULP Stream. The layer performing the RDMAP layer is not a ULP, but will likely have additional, local, functionality beyond the function performed by the RDMAP. For example, if the RDMAP (and lower) layer(s) are implemented on an RNIC, the RNIC may have additional error codes that are not captured by the RDMAP layer error codes currently in the specification, such as an error associated with an invalid state that is internal to the RNIC. These additional error codes would convey diagnostic/debug information that would be useful at both ends of the RDMA Stream being terminated. I would like to propose that we use value 1111b of the terminate layer field for RNIC related error types and codes. I would also like to propose that we state the error type and error code fields are reserved for use by the RNIC when the layer is 1111b. Thanks, Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 7 11:01:54 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA14176 for ; Fri, 7 Feb 2003 11:01:54 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h17G8rO12711 for rddp-archive@odin.ietf.org; Fri, 7 Feb 2003 11:08:53 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h17G8rp12708 for ; Fri, 7 Feb 2003 11:08:53 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA14163 for ; Fri, 7 Feb 2003 11:01:23 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h17G8Bp12697; Fri, 7 Feb 2003 11:08:11 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h17G5Bp11970 for ; Fri, 7 Feb 2003 11:05:11 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA14058 for ; Fri, 7 Feb 2003 10:57:40 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id IAA11634 for ; Fri, 7 Feb 2003 08:01:17 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Fri, 7 Feb 2003 11:01:16 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FE1@xbl.ma.emulex.com> From: "Williams, Jim" To: "'rddp@ietf.org'" Date: Fri, 7 Feb 2003 11:01:08 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] STag Invalidation Question Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , I am sure this issue has been discussed and debated before although perhaps not in this forum. I would be interested in hearing some of the thoughts here, though. PROPOSAL: STags are always invalidated immediately on on completion of RDDP writes. This is done before delivery of the data to the application. Therefore all tagged buffers are "write once" and must be re-registered before they can be written again. There are some interesting advantages to this approach. 1. It goes a long way towards resolving some of the issues raised in http://www.ietf.org/internet-drafts/draft-ietf-rddp-rdma-concerns-00.txt Particularly section 4.1 "Overwrites", and 4.2 "Concurrent Operations to the Same Memory" 2. It eliminates the need for "remote STag invalidate" which strikes me as an undesirable complexity in the currently proposed protocol definition. 3. It allows speculative out-of-order placement without the need for periodic markers or absolute alignment guarantees. Specifically, if incorrect placement is done, it can always be caught before delivering to the application, and is therefore harmless. Incorrect placement can (based on some assumptions a bit too complex to describe here) be transparently fixed by falling back to in order placement and overwriting the bad data with good data. #3 is complex enough to deserve a draft to fully explore, however, some initial discussion would be worth while to gage whether this path is worth going down. No doubt there are some good arguments for the current proposal and against mandating "one shot" tagged buffers. I would like to hear the justification. - Jim Williams _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 7 12:08:57 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA15700 for ; Fri, 7 Feb 2003 12:08:56 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h17HFvn16828 for rddp-archive@odin.ietf.org; Fri, 7 Feb 2003 12:15:57 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h17HFvp16825 for ; Fri, 7 Feb 2003 12:15:57 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA15689 for ; Fri, 7 Feb 2003 12:08:25 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h17HC7p16668; Fri, 7 Feb 2003 12:12:07 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h17HBGp16618 for ; Fri, 7 Feb 2003 12:11:16 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA15591 for ; Fri, 7 Feb 2003 12:03:44 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 42D462447F1; Fri, 7 Feb 2003 12:07:18 -0500 (EST) Date: Fri, 7 Feb 2003 11:08:23 -0600 From: Caitlin Bestler Subject: Re: [rddp] STag Invalidation Question To: "Williams, Jim" , rddp@ietf.org X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B8FE1@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/7/03, Williams, Jim wrote: > >I am sure this issue has been discussed and debated before >although perhaps not in this forum. I would be interested >in hearing some of the thoughts here, though. > >PROPOSAL: STags are always invalidated immediately on > on completion of RDDP writes. This is done > before delivery of the data to the application. > Therefore all tagged buffers are "write once" > and must be re-registered before they can > be written again. > There are numerous reasons for wanting to re-use STags that relate to the presumption that binding them for each operation will be too expensive. I fully agree that those reasons are insufficient. Implementations SHOULD assure that binding and invalidating an STag for each RDMA Write is a truly valid option. No ULP should refrain from doing so for fear of performance penalties. Numerous algorithms developed for dual-ported memory are applicable to RDMA and use of persistent STags. The primary example for me remains Variable Bit Rate (VBR) delivery. The Data Sink advertises a single large buffer intended to accept data over an extended period of time. Sends will be sent at timed intervals to confirm how much data has been placed (and allowing it to be deliverd). Because the amount of data transferred per completion is variable, there is no way to pre-feed future STags. The starting address for pipeline stage N begins *after* stage N-1, and this would not have been known when a distinct STag would have had to have been created. Another example would be a SAN or NAS server that has been asked to write 100 MB from a client buffer specified with an STag and TO. Unless it can dedicate a 100 MB buffer (unlikely), the server will be using RDMA Reads to fetch chunks matching half of its buffer size (perhaps 1 MB), and then writing from those buffer to disk. (Two buffers wouuld be used at the minimum so that one can be pre-loaded while the other is written to disk). These have to be distinct RDMA Reads, so that a completion is generated for each portion. The completion would trigger the actual write to disk. It is outside the scope of the IETF, but I would certainly recomend that applications SHOULD use one-shot STags, and that support for invalidation SHOULD be efficient enough so that no ULP would avoid one-shot STags for performance reasons. The applications that have need to use persistent STags have valid reasons of the type needed to override a SHOULD. They are advanced techniques that should not be selected by default. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 7 14:19:51 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA18840 for ; Fri, 7 Feb 2003 14:19:51 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h17JQtP24215 for rddp-archive@odin.ietf.org; Fri, 7 Feb 2003 14:26:55 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h17JQtp24212 for ; Fri, 7 Feb 2003 14:26:55 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA18815 for ; Fri, 7 Feb 2003 14:19:20 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h17JQDp24192; Fri, 7 Feb 2003 14:26:13 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h17JPhp24159 for ; Fri, 7 Feb 2003 14:25:43 -0500 Received: from ztxmail04.ztx.compaq.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA18786 for ; Fri, 7 Feb 2003 14:18:07 -0500 (EST) Received: from cceexg13.americas.cpqcorp.net (cceexg13.americas.cpqcorp.net [16.110.250.119]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id 46F0F2A15 for ; Fri, 7 Feb 2003 13:21:47 -0600 (CST) Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg13.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Fri, 7 Feb 2003 13:21:47 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: RE: [rddp] STag Invalidation Question Date: Fri, 7 Feb 2003 13:21:46 -0600 Message-ID: <4D027986353D1341ADA4F2A4F8A170762975A1DE@cceexc18.americas.cpqcorp.net> Thread-Topic: [rddp] STag Invalidation Question Thread-Index: AcLOzIXfEGggMPBtRPuEnh3g3UBLZwADm2SQ From: "Culley, Paul" To: X-OriginalArrivalTime: 07 Feb 2003 19:21:47.0144 (UTC) FILETIME=[28261080:01C2CEDE] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h17JPhp24160 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit I agree with Caitlin, there are lots of applications that want to open up a potentially large buffer for access by a remote agent; that remote agent will often want to perform multiple access, perhaps even involving writes and reads to that buffer before it gets invalidated. The Send+Invalidate allows the remote side to perform the invalidate when all such access are completed (usually when the ULP operation has been satisfied and when returning a status to the local ULP). Although not specified in the RDMAP specification, the local ULP should be informed when or if the Invalidate was done, so it can cross check that it has been done. Most folks in the original author team thought that the difference in time/security between getting a "one-shot" invalidation, and invalidating by the ULP when the ULP has been told that the buffers are no longer being used remotely, to be small enough that requiring the "one-shot" functionality was not worth it. Paul R. Culley HP Senior Fellow 281-514-5543 -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Friday, February 07, 2003 11:08 AM To: Williams, Jim; rddp@ietf.org Subject: Re: [rddp] STag Invalidation Question On 2/7/03, Williams, Jim wrote: > >I am sure this issue has been discussed and debated before although >perhaps not in this forum. I would be interested in hearing some of >the thoughts here, though. > >PROPOSAL: STags are always invalidated immediately on > on completion of RDDP writes. This is done > before delivery of the data to the application. > Therefore all tagged buffers are "write once" > and must be re-registered before they can > be written again. > There are numerous reasons for wanting to re-use STags that relate to the presumption that binding them for each operation will be too expensive. I fully agree that those reasons are insufficient. Implementations SHOULD assure that binding and invalidating an STag for each RDMA Write is a truly valid option. No ULP should refrain from doing so for fear of performance penalties. Numerous algorithms developed for dual-ported memory are applicable to RDMA and use of persistent STags. The primary example for me remains Variable Bit Rate (VBR) delivery. The Data Sink advertises a single large buffer intended to accept data over an extended period of time. Sends will be sent at timed intervals to confirm how much data has been placed (and allowing it to be deliverd). Because the amount of data transferred per completion is variable, there is no way to pre-feed future STags. The starting address for pipeline stage N begins *after* stage N-1, and this would not have been known when a distinct STag would have had to have been created. Another example would be a SAN or NAS server that has been asked to write 100 MB from a client buffer specified with an STag and TO. Unless it can dedicate a 100 MB buffer (unlikely), the server will be using RDMA Reads to fetch chunks matching half of its buffer size (perhaps 1 MB), and then writing from those buffer to disk. (Two buffers wouuld be used at the minimum so that one can be pre-loaded while the other is written to disk). These have to be distinct RDMA Reads, so that a completion is generated for each portion. The completion would trigger the actual write to disk. It is outside the scope of the IETF, but I would certainly recomend that applications SHOULD use one-shot STags, and that support for invalidation SHOULD be efficient enough so that no ULP would avoid one-shot STags for performance reasons. The applications that have need to use persistent STags have valid reasons of the type needed to override a SHOULD. They are advanced techniques that should not be selected by default. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 7 19:23:46 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA25981 for ; Fri, 7 Feb 2003 19:23:46 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h180UuE08618 for rddp-archive@odin.ietf.org; Fri, 7 Feb 2003 19:30:56 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h180Uup08615 for ; Fri, 7 Feb 2003 19:30:56 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA25952 for ; Fri, 7 Feb 2003 19:23:14 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h180UDp08573; Fri, 7 Feb 2003 19:30:13 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h180Tgp08525 for ; Fri, 7 Feb 2003 19:29:42 -0500 Received: from msgbas2x.cos.agilent.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA25911 for ; Fri, 7 Feb 2003 19:22:00 -0500 (EST) From: pat_thaler@agilent.com Received: from relcos2.cos.agilent.com (relcos2.cos.agilent.com [130.29.152.237]) by msgbas2x.cos.agilent.com (Postfix) with ESMTP id 14E661228; Fri, 7 Feb 2003 17:25:40 -0700 (MST) Received: from axcsbh4.cos.agilent.com (axcsbh4.cos.agilent.com [130.29.152.145]) by relcos2.cos.agilent.com (Postfix) with SMTP id AC23245C; Fri, 7 Feb 2003 17:25:39 -0700 (MST) Received: from 130.29.152.145 by axcsbh4.cos.agilent.com (InterScan E-Mail VirusWall NT); Fri, 07 Feb 2003 17:25:38 -0700 Received: by axcsbh4.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Fri, 7 Feb 2003 17:25:38 -0700 Message-ID: <1BEBA5E8600DD4119A50009027AF54A010CA027D@axcs04.cos.agilent.com> To: cait@asomi.com, Jim.Williams@emulex.com, rddp@ietf.org Subject: RE: [rddp] STag Invalidation Question Date: Fri, 7 Feb 2003 17:25:37 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Caitlin, There are reasons for wanting to do multiple RDMA operations to an advertised buffer that have nothing to do with the expense of binding to the STag. These reasons often have to do with letting the client and the server have independent control of the amount of buffering they use to support transfers. Your write of a file from a client to a server is one example of this. Another example is the case where a client ULP requests its server ULP to write a file. The client sends an STag in the request to advertise the buffer for the file. The server may not have all the data ready for the transfer at once and the client shouldn't have to know that detail in order to set up its buffer. When the server ULP has data ready to transfer to the client, it will start an RDMA Write for that data. Note that an RDMAP receiving RDMA Writes does not deliver them the ULP. Whether the request takes one RDMA Write or many is transparent to the client ULP. When all the RDMA Writes necessary to transfer the data have been initiated, the server ULP can send an RDMA Send with Invalidate message to invalidate the STag and carry a ULP PDU that informs the client ULP that the request has been fulfilled. This send may also have other information that the ULP will need in order to read the data such as the length of data that was actually written if the response was not a fixed size. Since all prior RDMA Writes must have completed before the RDMA Send message is delivered to the ULP, receipt of the Send ensures that the data is in the advertised buffer. Invalidating STags on RDMA Writes as Jim suggests would be incompatible with the operation above and it is entirely unnecessary. Since RDMA Writes are not surfaced to the ULP, it is the Send which tells the ULP that it can use the Data. Therefore, the Send with Invalidate is the tool that supplies the control that Jim wants. Regards, Pat -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Friday, February 07, 2003 9:08 AM To: Williams, Jim; rddp@ietf.org Subject: Re: [rddp] STag Invalidation Question On 2/7/03, Williams, Jim wrote: > >I am sure this issue has been discussed and debated before >although perhaps not in this forum. I would be interested >in hearing some of the thoughts here, though. > >PROPOSAL: STags are always invalidated immediately on > on completion of RDDP writes. This is done > before delivery of the data to the application. > Therefore all tagged buffers are "write once" > and must be re-registered before they can > be written again. > There are numerous reasons for wanting to re-use STags that relate to the presumption that binding them for each operation will be too expensive. I fully agree that those reasons are insufficient. Implementations SHOULD assure that binding and invalidating an STag for each RDMA Write is a truly valid option. No ULP should refrain from doing so for fear of performance penalties. Numerous algorithms developed for dual-ported memory are applicable to RDMA and use of persistent STags. The primary example for me remains Variable Bit Rate (VBR) delivery. The Data Sink advertises a single large buffer intended to accept data over an extended period of time. Sends will be sent at timed intervals to confirm how much data has been placed (and allowing it to be deliverd). Because the amount of data transferred per completion is variable, there is no way to pre-feed future STags. The starting address for pipeline stage N begins *after* stage N-1, and this would not have been known when a distinct STag would have had to have been created. Another example would be a SAN or NAS server that has been asked to write 100 MB from a client buffer specified with an STag and TO. Unless it can dedicate a 100 MB buffer (unlikely), the server will be using RDMA Reads to fetch chunks matching half of its buffer size (perhaps 1 MB), and then writing from those buffer to disk. (Two buffers wouuld be used at the minimum so that one can be pre-loaded while the other is written to disk). These have to be distinct RDMA Reads, so that a completion is generated for each portion. The completion would trigger the actual write to disk. It is outside the scope of the IETF, but I would certainly recomend that applications SHOULD use one-shot STags, and that support for invalidation SHOULD be efficient enough so that no ULP would avoid one-shot STags for performance reasons. The applications that have need to use persistent STags have valid reasons of the type needed to override a SHOULD. They are advanced techniques that should not be selected by default. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 7 19:48:36 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA26445 for ; Fri, 7 Feb 2003 19:48:35 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h180tkb09985 for rddp-archive@odin.ietf.org; Fri, 7 Feb 2003 19:55:46 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h180tkp09982 for ; Fri, 7 Feb 2003 19:55:46 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA26429 for ; Fri, 7 Feb 2003 19:48:04 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h180t4p09951; Fri, 7 Feb 2003 19:55:04 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h180sBp09917 for ; Fri, 7 Feb 2003 19:54:11 -0500 Received: from magic.adaptec.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA26415 for ; Fri, 7 Feb 2003 19:46:29 -0500 (EST) Received: from redfish.adaptec.com (redfish.adaptec.com [162.62.50.11]) by magic.adaptec.com (8.11.6+Sun/8.11.6) with ESMTP id h180o9D26469 for ; Fri, 7 Feb 2003 16:50:09 -0800 (PST) Received: from aimexc07.adaptec.com (aimexc07.adaptec.com [162.62.62.47]) by redfish.adaptec.com (8.8.8+Sun/8.8.8) with ESMTP id QAA21905 for ; Fri, 7 Feb 2003 16:50:08 -0800 (PST) Received: by aimexc07.adaptec.com with Internet Mail Service (5.5.2653.19) id ; Fri, 7 Feb 2003 16:50:08 -0800 Message-ID: From: "Carrier, John" To: rddp@ietf.org Date: Fri, 7 Feb 2003 16:50:08 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01C2CF0C.06D4C600" Subject: [rddp] FW: I-D ACTION:draft-elzur-iwarp-mpa-tcp-analysis-00.txt Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01C2CF0C.06D4C600 Content-Type: text/plain; charset="iso-8859-1" > -----Original Message----- > From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org] > Sent: Friday, February 07, 2003 5:07 AM > Subject: I-D ACTION:draft-elzur-iwarp-mpa-tcp-analysis-00.txt > > > A New Internet-Draft is available from the on-line > Internet-Drafts directories. > > > Title : Analysis of MPA over TCP Operations > Author(s) : U. Elzur > Filename : draft-elzur-iwarp-mpa-tcp-analysis-00.txt > Pages : 38 > Date : 2003-2-6 > > Further explanation and analysis of architectural recommendations > contained in the recent Internet-Draft, Marker PDU Aligned Framing > for TCP Specification [MPA], is provided. The impact of the > following three attributes of MPA over TCP is examined: > * packing of multiple DDP ULPDUs into one TCP segment; > * simplifying the receiver due to transmitter alignment of DDP > headers with TCP headers; and > * mistakenly attempting to interoperate an MPA-enabled endpoint > with a non-MPA-enabled endpoint. > > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-elzur-iwarp-mpa-tcp- analysis-00.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-elzur-iwarp-mpa-tcp-analysis-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-elzur-iwarp-mpa-tcp-analysis-00.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ------_=_NextPart_000_01C2CF0C.06D4C600 Content-Type: message/rfc822 To: Subject: Date: Fri, 7 Feb 2003 11:47:28 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: multipart/mixed; boundary="----_=_NextPart_002_01C2CF0C.06D4C600" ------_=_NextPart_002_01C2CF0C.06D4C600 Content-Type: text/plain ------_=_NextPart_002_01C2CF0C.06D4C600 Content-Type: application/octet-stream; name="ATT31999" Content-Disposition: attachment; filename="ATT31999" Content-type: message/external-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <2003-2-6152424.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-elzur-iwarp-mpa-tcp-analysis-00.txt ------_=_NextPart_002_01C2CF0C.06D4C600 Content-Type: message/external-body; site="internet-drafts"; dir="draft-elzur-iwarp-mpa-tcp-analysis-00.txt"; mode="ftp.ietf.org"; access-type="anon-ftp" ------_=_NextPart_002_01C2CF0C.06D4C600-- ------_=_NextPart_000_01C2CF0C.06D4C600-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 7 20:57:46 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA27462 for ; Fri, 7 Feb 2003 20:57:46 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1824we12853 for rddp-archive@odin.ietf.org; Fri, 7 Feb 2003 21:04:58 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1824wp12850 for ; Fri, 7 Feb 2003 21:04:58 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA27450 for ; Fri, 7 Feb 2003 20:57:14 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1824Ap12832; Fri, 7 Feb 2003 21:04:10 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1823Up12789 for ; Fri, 7 Feb 2003 21:03:30 -0500 Received: from e33.co.us.ibm.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA27427; Fri, 7 Feb 2003 20:55:45 -0500 (EST) Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e33.co.us.ibm.com (8.12.7/8.12.2) with ESMTP id h181xMUn033012; Fri, 7 Feb 2003 20:59:22 -0500 Received: from d03nm014.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.3/NCO/VER6.5) with ESMTP id h181xLi0061888; Fri, 7 Feb 2003 18:59:21 -0700 Subject: RE: [rddp] STag Invalidation Question To: cait@asomi.com Cc: Jim.Williams@emulex.com, rddp@ietf.org, rddp-admin@ietf.org, pat_thaler@agilent.com X-Mailer: Lotus Notes Release 5.0.11 July 24, 2002 Message-ID: From: John Hufferd Date: Fri, 7 Feb 2003 17:56:11 -0800 X-MIMETrack: Serialize by Router on D03NM014/03/M/IBM(Release 6.0 [IBM]|December 16, 2002) at 02/07/2003 18:59:21 MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , I completely agree with Pat. She is describing the way a storage controller works with a host. It would make no sense for the storage controller to get additional STags, from the host, for each segment of I/O it is sending to the Host (The size and number of segments depends on the Buffering/Caching approach taken by the Storage Controller). Plus the host will not know what buffering and rotational delay, etc. is being handled by the Storage Controller so it will not know how many STags to send the Storage Controller for any specific I/O request. This may cause the Storage Controller to ask the host to send additional STags, in the middle of the I/O transfer, so that it can finish the requested I/O. (Not a very good design.) . . . John L. Hufferd Senior Technical Staff Member (STSM) IBM/Systems Group, San Jose CA Main Office (408) 256-0403, Tie:276-0403, eFax: (408) 904-4688 Home Office: (408) 997-6136, Cell: (408) 499-9702 Internet Address: hufferd@us.ibm.com pat_thaler@agilen t.com To: cait@asomi.com, Jim.Williams@emulex.com, rddp@ietf.org Sent by: cc: rddp-admin@ietf.o Subject: RE: [rddp] STag Invalidation Question rg 02/07/2003 04:25 PM Caitlin, There are reasons for wanting to do multiple RDMA operations to an advertised buffer that have nothing to do with the expense of binding to the STag. These reasons often have to do with letting the client and the server have independent control of the amount of buffering they use to support transfers. Your write of a file from a client to a server is one example of this. Another example is the case where a client ULP requests its server ULP to write a file. The client sends an STag in the request to advertise the buffer for the file. The server may not have all the data ready for the transfer at once and the client shouldn't have to know that detail in order to set up its buffer. When the server ULP has data ready to transfer to the client, it will start an RDMA Write for that data. Note that an RDMAP receiving RDMA Writes does not deliver them the ULP. Whether the request takes one RDMA Write or many is transparent to the client ULP. When all the RDMA Writes necessary to transfer the data have been initiated, the server ULP can send an RDMA Send with Invalidate message to invalidate the STag and carry a ULP PDU that informs the client ULP that the request has been fulfilled. This send may also have other information that the ULP will need in order to read the data such as the length of data that was actually written if the response was not a fixed size. Since all prior RDMA Writes must have completed before the RDMA Send message is delivered to the ULP, receipt of the Send ensures that the data is in the advertised buffer. Invalidating STags on RDMA Writes as Jim suggests would be incompatible with the operation above and it is entirely unnecessary. Since RDMA Writes are not surfaced to the ULP, it is the Send which tells the ULP that it can use the Data. Therefore, the Send with Invalidate is the tool that supplies the control that Jim wants. Regards, Pat -----Original Message----- From: Caitlin Bestler [mailto:cait@asomi.com] Sent: Friday, February 07, 2003 9:08 AM To: Williams, Jim; rddp@ietf.org Subject: Re: [rddp] STag Invalidation Question On 2/7/03, Williams, Jim wrote: > >I am sure this issue has been discussed and debated before >although perhaps not in this forum. I would be interested >in hearing some of the thoughts here, though. > >PROPOSAL: STags are always invalidated immediately on > on completion of RDDP writes. This is done > before delivery of the data to the application. > Therefore all tagged buffers are "write once" > and must be re-registered before they can > be written again. > There are numerous reasons for wanting to re-use STags that relate to the presumption that binding them for each operation will be too expensive. I fully agree that those reasons are insufficient. Implementations SHOULD assure that binding and invalidating an STag for each RDMA Write is a truly valid option. No ULP should refrain from doing so for fear of performance penalties. Numerous algorithms developed for dual-ported memory are applicable to RDMA and use of persistent STags. The primary example for me remains Variable Bit Rate (VBR) delivery. The Data Sink advertises a single large buffer intended to accept data over an extended period of time. Sends will be sent at timed intervals to confirm how much data has been placed (and allowing it to be deliverd). Because the amount of data transferred per completion is variable, there is no way to pre-feed future STags. The starting address for pipeline stage N begins *after* stage N-1, and this would not have been known when a distinct STag would have had to have been created. Another example would be a SAN or NAS server that has been asked to write 100 MB from a client buffer specified with an STag and TO. Unless it can dedicate a 100 MB buffer (unlikely), the server will be using RDMA Reads to fetch chunks matching half of its buffer size (perhaps 1 MB), and then writing from those buffer to disk. (Two buffers wouuld be used at the minimum so that one can be pre-loaded while the other is written to disk). These have to be distinct RDMA Reads, so that a completion is generated for each portion. The completion would trigger the actual write to disk. It is outside the scope of the IETF, but I would certainly recomend that applications SHOULD use one-shot STags, and that support for invalidation SHOULD be efficient enough so that no ULP would avoid one-shot STags for performance reasons. The applications that have need to use persistent STags have valid reasons of the type needed to override a SHOULD. They are advanced techniques that should not be selected by default. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 7 23:58:42 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA01663 for ; Fri, 7 Feb 2003 23:58:42 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1855vd20908 for rddp-archive@odin.ietf.org; Sat, 8 Feb 2003 00:05:57 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1855vp20905 for ; Sat, 8 Feb 2003 00:05:57 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA01650 for ; Fri, 7 Feb 2003 23:58:11 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1855Fp20894; Sat, 8 Feb 2003 00:05:15 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1854lp20859 for ; Sat, 8 Feb 2003 00:04:47 -0500 Received: from mxic1.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA01638 for ; Fri, 7 Feb 2003 23:57:01 -0500 (EST) From: Black_David@emc.com Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1KDYNVSF>; Sat, 8 Feb 2003 00:00:40 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C871@corpmx14.us.dg.com> To: rddp@ietf.org Date: Fri, 7 Feb 2003 23:59:02 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] Please review arch. draft Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Folks, According to our charter, the problem statement and architecture drafts should be completed in March. As of the Atlanta meeting, the problem statement draft was in good shape and could go to WG Last Call in its current form, but the architecture draft still needed more work. I would ask everyone to please review the current architecture draft: http://www.ietf.org/internet-drafts/draft-ietf-rddp-arch-00.txt and send comments/suggestions/etc. to the list or directly to Tom Talpey . Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 10 15:07:41 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA23275 for ; Mon, 10 Feb 2003 15:07:41 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1AKGDj08486 for rddp-archive@odin.ietf.org; Mon, 10 Feb 2003 15:16:13 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1AKGDp08483 for ; Mon, 10 Feb 2003 15:16:13 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA23244 for ; Mon, 10 Feb 2003 15:07:10 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1AKFNp08450; Mon, 10 Feb 2003 15:15:23 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1AKELp08414 for ; Mon, 10 Feb 2003 15:14:21 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA23203 for ; Mon, 10 Feb 2003 15:05:18 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id MAA03962 for ; Mon, 10 Feb 2003 12:08:58 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id ; Mon, 10 Feb 2003 15:08:57 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FE6@xbl.ma.emulex.com> From: "Williams, Jim" To: rddp@ietf.org Subject: RE: [rddp] STag Invalidation Question Date: Mon, 10 Feb 2003 15:08:56 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , pat_thaler wrote: > Your write of a file from a client to a server is one example > of this. Another example is the case where a client ULP > requests its server ULP to write a file. The client sends an > STag in the request to advertise the buffer for the file. The > server may not have all the data ready for the transfer at > once and the client shouldn't have to know that detail in > order to set up its buffer. When the server ULP has data > ready to transfer to the client, it will start an RDMA Write > for that data. Note that an RDMAP receiving RDMA Writes does > not deliver them the ULP. Whether the request takes one RDMA > Write or many is transparent to the client ULP. You make a good point. I think I agree with this. Need to think a bit more about the whole problem. - Jim _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 11 11:04:55 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA09296 for ; Tue, 11 Feb 2003 11:04:55 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1BGDqj24975 for rddp-archive@odin.ietf.org; Tue, 11 Feb 2003 11:13:52 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BGDqp24972 for ; Tue, 11 Feb 2003 11:13:52 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA09291 for ; Tue, 11 Feb 2003 11:04:24 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BGDBp24941; Tue, 11 Feb 2003 11:13:11 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BGCup24917 for ; Tue, 11 Feb 2003 11:12:56 -0500 Received: from e35.co.us.ibm.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA09272 for ; Tue, 11 Feb 2003 11:03:27 -0500 (EST) Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e35.co.us.ibm.com (8.12.7/8.12.2) with ESMTP id h1BG79Ko149836 for ; Tue, 11 Feb 2003 11:07:09 -0500 Received: from d03nm803.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.3/NCO/VER6.5) with ESMTP id h1BG5qrH084550 for ; Tue, 11 Feb 2003 09:05:53 -0700 To: rddp@ietf.org X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: Renato Recio Date: Tue, 11 Feb 2003 10:05:51 -0600 X-MIMETrack: Serialize by Router on D03NM803/03/M/IBM(Release 6.0 [IBM]|December 16, 2002) at 02/11/2003 09:05:52 MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Subject: [rddp] Association of STag and RDMAP Stream(s) Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Folks, A problem was raised on this reflector regarding the need for a little more text describing the mechanisms the RDMAP layer may use to associate an RDMAP Stream and an STag. Following is text I propose to add to bullet #5 in section 10.3 of the -01 specification: RDMAP MAY choose several locally defined mechanisms for associating a RDMAP Stream and a STag. One such mechanism is to provide two association types: a protection domain association and an RDMAP Stream association. Under the protection domain (PD) association, a unique protection domain identifier (PD ID) is created and used locally to associate the STag with a set of RDMAP Streams. Under this mechanism, only RDMAP Streams that have the same PD ID as the STag are allowed to use the STag. For an incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the STag targeted by the RDMA Write or RDMA Read Response Message (respectively), then the Message is not placed and the RDMAP Stream MUST be Terminated. For an incoming RDMA Read Request Message on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the Data Source STag, the Message is not placed and the RDMAP Stream MUST be Terminated. Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated. Note that the PD ID is locally defined, and cannot be directly manipulated by the Remote Peer. Under RDMAP Stream association, a given RDMAP Stream is identified locally by a unique RDMAP Stream identifier (ID) and that RDMA Stream ID is associated with the STag and RDMAP Stream. In this case, for an incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the STag targeted by the RDMA Write or RDMA Read Response Message (respectively), then the Message is not placed and the RDMAP Stream MUST be Terminated. Similarly, for an incoming RDMA Read Request Message on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the Data Source STag, then the Message is not placed and the RDMAP Stream MUST be Terminated. Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated. Note that the RDMA Stream ID is locally defined, and cannot be directly manipulated by the Remote Peer. I believe the above text resolves the problem, folks, please let me know if you agree. Thanks, Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 11 12:05:50 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA11638 for ; Tue, 11 Feb 2003 12:05:50 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1BHEnM29518 for rddp-archive@odin.ietf.org; Tue, 11 Feb 2003 12:14:49 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BHEnp29515 for ; Tue, 11 Feb 2003 12:14:49 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA11633 for ; Tue, 11 Feb 2003 12:05:19 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BHE9p29502; Tue, 11 Feb 2003 12:14:09 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BHDxp29485 for ; Tue, 11 Feb 2003 12:13:59 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA11620 for ; Tue, 11 Feb 2003 12:04:29 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id B6C99244B3E; Tue, 11 Feb 2003 12:08:09 -0500 (EST) Date: Tue, 11 Feb 2003 11:09:18 -0600 From: Caitlin Bestler Subject: Re: [rddp] Association of STag and RDMAP Stream(s) To: Renato Recio , rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/11/03, Renato Recio wrote: > > > > >Folks, > A problem was raised on this reflector regarding the need > for a little more text describing the mechanisms the > RDMAP > layer may use to associate an RDMAP Stream and an > STag. > > Following is text I propose to add to bullet #5 in > section 10.3 of the -01 specification: > > Two points: First, I believe that validation of STags is a DDP issue, not RDMAP. So the extra wording should go in *that* draft. STag Invalidation may be performed by RDMAP, but validation must be performed *before* placement, and therefore must be handled by the DDP layer. Second, I believe it should be made clear that this does not imply a standard scope for an STag. The naming domain obviously must be at least as large as the domain over which it is valid, but it could be only that size, the scope of the IP address, the subnet or whatever. If you use an STag somewhere else, you have no guarantee that you will not accidentally pick a valid value. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 11 13:14:55 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA13697 for ; Tue, 11 Feb 2003 13:14:55 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1BINuX01054 for rddp-archive@odin.ietf.org; Tue, 11 Feb 2003 13:23:56 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BINtp01051 for ; Tue, 11 Feb 2003 13:23:55 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA13690 for ; Tue, 11 Feb 2003 13:14:24 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BINAp01008; Tue, 11 Feb 2003 13:23:11 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BIMqp00985 for ; Tue, 11 Feb 2003 13:22:52 -0500 Received: from e32.co.us.ibm.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA13660; Tue, 11 Feb 2003 13:13:19 -0500 (EST) Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e32.co.us.ibm.com (8.12.7/8.12.2) with ESMTP id h1BIGu5V006538; Tue, 11 Feb 2003 13:16:56 -0500 Received: from d03nm803.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.3/NCO/VER6.5) with ESMTP id h1BIGprI052838; Tue, 11 Feb 2003 11:16:54 -0700 Subject: Re: [rddp] Association of STag and RDMAP Stream(s) To: Caitlin Bestler Cc: rddp@ietf.org, rddp-admin@ietf.org X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: Renato Recio Date: Tue, 11 Feb 2003 12:16:50 -0600 X-MIMETrack: Serialize by Router on D03NM803/03/M/IBM(Release 6.0 [IBM]|December 16, 2002) at 02/11/2003 11:16:54 MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Caitlin, My reponses bounded by . Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 Caitlin Bestler To: Renato Recio/Austin/IBM@IBMUS, rddp@ietf.org Sent by: cc: rddp-admin@ietf.o Subject: Re: [rddp] Association of STag and RDMAP Stream(s) rg 02/11/2003 11:09 AM On 2/11/03, Renato Recio wrote: > > > > >Folks, > A problem was raised on this reflector regarding the need > for a little more text describing the mechanisms the > RDMAP > layer may use to associate an RDMAP Stream and an > STag. > > Following is text I propose to add to bullet #5 in > section 10.3 of the -01 specification: > > Two points: First, I believe that validation of STags is a DDP issue, not RDMAP. So the extra wording should go in *that* draft. In my view, validation of STags is performed by DDP for RDMA Writes and RDMA Read Responses. It is performed by RDMA for RDMA Read Requests, Send with Invalidate, and Send with SE and Invalidate. I think the RDMA text should span all messages that perform STag validation. However, I agree the text should span the two specifications. I'll offer the suggested text shortly. STag Invalidation may be performed by RDMAP, but validation must be performed *before* placement, and therefore must be handled by the DDP layer. Agreed for RDMA Writes and RDMA Read Responses. However, DDP is unaware that Untagged Messages carry an STag and cannot do STag validation for RDMA Read Requests, Send with Invalidate, and Send with SE and Invalidate. For these messages RDMA must perform STag validation. Second, I believe it should be made clear that this does not imply a standard scope for an STag. The naming domain obviously must be at least as large as the domain over which it is valid, but it could be only that size, the scope of the IP address, the subnet or whatever. The additional text does not mention the scope of an STag and I agree it should not. Though the STag scope is dependent on the mechanism used to associate the STag to an RDMAP Stream and two such mechanisms are described in my e-mail, the RDMAP layer is free to choose whether to support one, both, or neither of the alternatives I described. If you use an STag somewhere else, you have no guarantee that you will not accidentally pick a valid value. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 11 13:43:01 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA14781 for ; Tue, 11 Feb 2003 13:43:00 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1BIq2q03075 for rddp-archive@odin.ietf.org; Tue, 11 Feb 2003 13:52:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BIq1p03072 for ; Tue, 11 Feb 2003 13:52:01 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA14756 for ; Tue, 11 Feb 2003 13:42:29 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BIo1p02988; Tue, 11 Feb 2003 13:50:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BInDp02954 for ; Tue, 11 Feb 2003 13:49:13 -0500 Received: from e32.co.us.ibm.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA14678 for ; Tue, 11 Feb 2003 13:39:40 -0500 (EST) Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e32.co.us.ibm.com (8.12.7/8.12.2) with ESMTP id h1BIhG5V011356 for ; Tue, 11 Feb 2003 13:43:19 -0500 Received: from d03nm803.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.3/NCO/VER6.5) with ESMTP id h1BIhCrH028958 for ; Tue, 11 Feb 2003 11:43:13 -0700 To: rddp@ietf.org X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: Renato Recio Date: Tue, 11 Feb 2003 12:43:10 -0600 X-MIMETrack: Serialize by Router on D03NM803/03/M/IBM(Release 6.0 [IBM]|December 16, 2002) at 02/11/2003 11:43:13 MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Subject: [rddp] Association of STag and RDMAP Stream(s) - version 2 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Folks, I believe the following text addresses the issue raised by Caitlin regarding STag validation. Following is text I propose to add to bullet #3 in section 10.3 of the DDP -01 specification: DDP MAY choose several locally defined mechanisms for associating a DDP Stream and a STag. One such mechanism is to provide two association types: a protection domain association and an DDP Stream association. Under the protection domain (PD) association, a unique protection domain identifier (PD ID) is created and used locally to associate the STag with a set of DDP Streams. Under this mechanism, only DDP Streams that have the same PD ID as the STag are allowed to use the STag. For an incoming Tagged Message on an DDP Stream, if the PD ID associated with that DDP Stream is not the same as the PD ID associated with the STag targeted by the Tagged Message, then the Message is not placed and the DDP Layer MUST surface a local error to the ULP. Note that the PD ID is locally defined, and cannot be directly manipulated by the Remote Peer. Under DDP Stream association, a given DDP Stream is identified locally by a unique DDP Stream identifier (ID) and that DDP Stream ID is associated with the STag and DDP Stream. In this case, for an incoming Tagged Message on an DDP Stream, if the DDP Stream ID associated with that DDP Stream is not the same as the DDP Stream ID associated with the STag targeted by the Tagged Message, then the Message is not placed and and the DDP Layer MUST surface a local error to the ULP. Note that the DDP Stream ID is locally defined, and cannot be directly manipulated by the Remote Peer. Following is text I propose to add to bullet #5 in section 10.3 of the -01 specification: RDMAP MAY choose several locally defined mechanisms for associating a RDMAP Stream and a STag. One such mechanism is to provide two association types: a protection domain association and an RDMAP Stream association. Under the protection domain (PD) association, a unique protection domain identifier (PD ID) is created and used locally to associate the STag with a set of RDMAP Streams. Under this mechanism, only RDMAP Streams that have the same PD ID as the STag are allowed to use the STag. For an incoming RDMA Read Request Message on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the Data Source STag, the Message is not placed and the RDMAP Stream MUST be Terminated with an Invalid STag error. Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated with an STag cannot be Invalidated error. Note that the PD ID is locally defined, and cannot be directly manipulated by the Remote Peer. Under RDMAP Stream association, a given RDMAP Stream is identified locally by a unique RDMAP Stream identifier (ID) and that RDMA Stream ID is associated with the STag and RDMAP Stream. For an incoming RDMA Read Request Message on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the Data Source STag, then the Message is not placed and the RDMAP Stream MUST be Terminated with an Invalid STag error. Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated with an STag cannot be Invalidated error. Note that the RDMA Stream ID is locally defined, and cannot be directly manipulated by the Remote Peer. Note: For an incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream, the DDP layer MUST associate the STag targeted by the RDMA Write or RDMA Read Response Message (respectively) to the RDMAP Stream. If an STag targeted by the RDMA Write or RDMA Read Response Message Segment is not associated with the RDMAP Stream that received the Message Segment, DDP MUST surface an Invalid STag error to the RDMAP layer. The RDMAP layer MUST Terminate the RDMAP Stream if the DDP Layer surfaces an Invalid STag error. I believe the above text resolves the problem Caitlin surfaced, folks, please let me know if you agree. Also, please note, the STag cannot be Invalidated error is one of the two new error codes I surfaced on this reflector. Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 11 14:00:18 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA15288 for ; Tue, 11 Feb 2003 14:00:18 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1BJ9Ku04515 for rddp-archive@odin.ietf.org; Tue, 11 Feb 2003 14:09:20 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BJ9Kp04512 for ; Tue, 11 Feb 2003 14:09:20 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA15261 for ; Tue, 11 Feb 2003 13:59:47 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BJ83p04446; Tue, 11 Feb 2003 14:08:03 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BJ7op04417 for ; Tue, 11 Feb 2003 14:07:50 -0500 Received: from e32.co.us.ibm.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA15238 for ; Tue, 11 Feb 2003 13:58:16 -0500 (EST) Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.194.23]) by e32.co.us.ibm.com (8.12.7/8.12.2) with ESMTP id h1BJ1w5V044688 for ; Tue, 11 Feb 2003 14:01:58 -0500 Received: from d03nm803.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay02.boulder.ibm.com (8.12.3/NCO/VER6.5) with ESMTP id h1BJ1uLg026806 for ; Tue, 11 Feb 2003 12:01:57 -0700 To: rddp@ietf.org X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: Renato Recio Date: Tue, 11 Feb 2003 13:01:54 -0600 X-MIMETrack: Serialize by Router on D03NM803/03/M/IBM(Release 6.0 [IBM]|December 16, 2002) at 02/11/2003 12:01:57 MIME-Version: 1.0 Content-type: text/plain; charset=US-ASCII Subject: [rddp] Association of STag and RDMAP Stream(s) - version 3 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Sorry folks, this is a resend, the second set of text is for the RDMA spec. I believe the following text addresses the issue raised by Caitlin regarding STag validation. Following is text I propose to add to bullet #3 in section 10.3 of the DDP -01 specification: DDP MAY choose several locally defined mechanisms for associating a DDP Stream and a STag. One such mechanism is to provide two association types: a protection domain association and an DDP Stream association. Under the protection domain (PD) association, a unique protection domain identifier (PD ID) is created and used locally to associate the STag with a set of DDP Streams. Under this mechanism, only DDP Streams that have the same PD ID as the STag are allowed to use the STag. For an incoming Tagged Message on an DDP Stream, if the PD ID associated with that DDP Stream is not the same as the PD ID associated with the STag targeted by the Tagged Message, then the Message is not placed and the DDP Layer MUST surface a local error to the ULP. Note that the PD ID is locally defined, and cannot be directly manipulated by the Remote Peer. Under DDP Stream association, a given DDP Stream is identified locally by a unique DDP Stream identifier (ID) and that DDP Stream ID is associated with the STag and DDP Stream. In this case, for an incoming Tagged Message on an DDP Stream, if the DDP Stream ID associated with that DDP Stream is not the same as the DDP Stream ID associated with the STag targeted by the Tagged Message, then the Message is not placed and and the DDP Layer MUST surface a local error to the ULP. Note that the DDP Stream ID is locally defined, and cannot be directly manipulated by the Remote Peer. Following is text I propose to add to bullet #5 in section 10.3 of the RDMAP -01 specification: RDMAP MAY choose several locally defined mechanisms for associating a RDMAP Stream and a STag. One such mechanism is to provide two association types: a protection domain association and an RDMAP Stream association. Under the protection domain (PD) association, a unique protection domain identifier (PD ID) is created and used locally to associate the STag with a set of RDMAP Streams. Under this mechanism, only RDMAP Streams that have the same PD ID as the STag are allowed to use the STag. For an incoming RDMA Read Request Message on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the Data Source STag, the Message is not placed and the RDMAP Stream MUST be Terminated with an Invalid STag error. Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated with an STag cannot be Invalidated error. Note that the PD ID is locally defined, and cannot be directly manipulated by the Remote Peer. Under RDMAP Stream association, a given RDMAP Stream is identified locally by a unique RDMAP Stream identifier (ID) and that RDMA Stream ID is associated with the STag and RDMAP Stream. For an incoming RDMA Read Request Message on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the Data Source STag, then the Message is not placed and the RDMAP Stream MUST be Terminated with an Invalid STag error. Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated with an STag cannot be Invalidated error. Note that the RDMA Stream ID is locally defined, and cannot be directly manipulated by the Remote Peer. Note: For an incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream, the DDP layer MUST associate the STag targeted by the RDMA Write or RDMA Read Response Message (respectively) to the RDMAP Stream. If an STag targeted by the RDMA Write or RDMA Read Response Message Segment is not associated with the RDMAP Stream that received the Message Segment, DDP MUST surface an Invalid STag error to the RDMAP layer. The RDMAP layer MUST Terminate the RDMAP Stream if the DDP Layer surfaces an Invalid STag error. I believe the above text resolves the problem Caitlin surfaced, folks, please let me know if you agree. Also, please note, the STag cannot be Invalidated error is one of the two new error codes I surfaced on this reflector. Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 11 15:48:21 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA18067 for ; Tue, 11 Feb 2003 15:48:21 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1BKvOI11242 for rddp-archive@odin.ietf.org; Tue, 11 Feb 2003 15:57:24 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BKvNp11239 for ; Tue, 11 Feb 2003 15:57:23 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA18063 for ; Tue, 11 Feb 2003 15:47:49 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BKuNp11214; Tue, 11 Feb 2003 15:56:23 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1BKtFp11167 for ; Tue, 11 Feb 2003 15:55:15 -0500 Received: from hermes.fm.intel.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA18030 for ; Tue, 11 Feb 2003 15:45:40 -0500 (EST) Received: from talaria.fm.intel.com (talaria.fm.intel.com [10.1.192.39]) by hermes.fm.intel.com (8.11.6/8.11.6/d: outer.mc,v 1.51 2002/09/23 20:43:23 dmccart Exp $) with ESMTP id h1BKkND28661 for ; Tue, 11 Feb 2003 20:46:24 GMT Received: from fmsmsxvs043.fm.intel.com (fmsmsxvs043.fm.intel.com [132.233.42.129]) by talaria.fm.intel.com (8.11.6/8.11.6/d: inner.mc,v 1.28 2003/01/13 19:44:39 dmccart Exp $) with SMTP id h1BKp7111369 for ; Tue, 11 Feb 2003 20:51:07 GMT Received: from fmsmsx331-2.fm.intel.com ([132.233.42.156]) by fmsmsxvs043.fm.intel.com (NAVGW 2.5.2.11) with SMTP id M2003021112472207236 ; Tue, 11 Feb 2003 12:47:22 -0800 Received: from fmsmsx402.amr.corp.intel.com ([132.233.42.200]) by fmsmsx331-2.fm.intel.com with Microsoft SMTPSVC(5.0.2195.5329); Tue, 11 Feb 2003 12:49:21 -0800 content-class: urn:content-classes:message Subject: RE: [rddp] Association of STag and RDMAP Stream(s) - version 3 Date: Tue, 11 Feb 2003 12:49:21 -0800 Message-ID: <6315617889C99D4BA7C14687DEC8DB4E02560331@fmsmsx402.fm.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Thread-Topic: [rddp] Association of STag and RDMAP Stream(s) - version 3 X-MimeOLE: Produced By Microsoft Exchange V6.0.6334.0 Thread-Index: AcLSACY87urn6T3yEdembgACpYtLYQABFYng From: "Shah, Hemal" To: "Renato Recio" , X-OriginalArrivalTime: 11 Feb 2003 20:49:21.0165 (UTC) FILETIME=[0D710BD0:01C2D20F] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h1BKtGp11168 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit Renato, I like the text you have for DDP and RDMAP. See my comments bounded by . Hemal -----Original Message----- From: Renato Recio [mailto:recio@us.ibm.com] Sent: Tuesday, February 11, 2003 1:02 PM To: rddp@ietf.org Subject: [rddp] Association of STag and RDMAP Stream(s) - version 3 Sorry folks, this is a resend, the second set of text is for the RDMA spec. I believe the following text addresses the issue raised by Caitlin regarding STag validation. Following is text I propose to add to bullet #3 in section 10.3 of the DDP -01 specification: DDP MAY choose several locally defined mechanisms for associating a DDP Stream and a STag. One such mechanism is to provide two association types: a protection domain association and an DDP Stream association. I would rephrase the above paragraph as: There are several mechanisms for associating an STag and a DDP Stream. A couple of reasonable mechanisms for this association are: a protection domain association and a DDP Stream association. Under the protection domain (PD) association, a unique protection domain identifier (PD ID) is created and used locally to associate the STag with a set of DDP Streams. Under this mechanism, only DDP Streams that have the same PD ID as the STag are allowed to use the STag. For an incoming Tagged Message on an DDP Stream, if the PD ID associated with that DDP Stream is not the same as the PD ID associated with the STag targeted by the Tagged Message, then the Message is not placed and the DDP Layer MUST surface a local error to the ULP. Note that the PD ID is locally defined, and cannot be directly manipulated by the Remote Peer. I would rephrase the above paragraph (with slight modifications) as: Under the protection domain (PD) association, a unique protection domain identifier (PD ID) is created and used locally to associate an STag with a set of DDP Streams. Under this mechanism, the use of the STag is only permitted on the DDP Streams that have the same PD ID as the STag. For an incoming DDP Segment of a Tagged DDP Message on a DDP Stream, if the PD ID of the DDP Stream is not the same as the PD ID of the STag targeted by the Tagged DDP Message, then the DDP Segment is not placed and the DDP Layer MUST surface a local error to the ULP. Note that the PD ID is locally defined, and cannot be directly manipulated by the Remote Peer. Under DDP Stream association, a given DDP Stream is identified locally by a unique DDP Stream identifier (ID) and that DDP Stream ID is associated with the STag and DDP Stream. In this case, for an incoming Tagged Message on an DDP Stream, if the DDP Stream ID associated with that DDP Stream is not the same as the DDP Stream ID associated with the STag targeted by the Tagged Message, then the Message is not placed and and the DDP Layer MUST surface a local error to the ULP. Note that the DDP Stream ID is locally defined, and cannot be directly manipulated by the Remote Peer. I would rephrase the above paragraph (with slight modifications) as: Under DDP Stream association, a DDP Stream is identified locally by a unique DDP Stream identifier (ID). An STag is associated with a DDP Stream by using DDP Stream ID. In this case, for an incoming DDP Segment of a Tagged DDP Message on a DDP Stream, if the DDP Stream ID of the DDP Stream is not the same as the DDP Stream ID of the STag targeted by the Tagged DDP Message, then the DDP Segment is not placed and and the DDP Layer MUST surface a local error to the ULP. Note that the DDP Stream ID is locally defined, and cannot be directly manipulated by the Remote Peer. I will also add the following paragraph to make a strong requirement about the association of an STag and a DDP Stream: ULP SHOULD associate an STag and a DDP Stream. DDP MUST support protection domain association and DDP Stream association mechanisms for associating an STag and a DDP Stream. Following is text I propose to add to bullet #5 in section 10.3 of the RDMAP -01 specification: RDMAP MAY choose several locally defined mechanisms for associating a RDMAP Stream and a STag. One such mechanism is to provide two association types: a protection domain association and an RDMAP Stream association. Under the protection domain (PD) association, a unique protection domain identifier (PD ID) is created and used locally to associate the STag with a set of RDMAP Streams. Under this mechanism, only RDMAP Streams that have the same PD ID as the STag are allowed to use the STag. For an incoming RDMA Read Request Message on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the Data Source STag, the Message is not placed and the RDMAP Stream MUST be Terminated with an Invalid STag error. Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated with an STag cannot be Invalidated Error. Note that the PD ID is locally defined, and cannot be directly manipulated by the Remote Peer. In case of SendInv and SendSEInv, when you check for the STag association in RDMAP layer, the Untagged DDP Message has been already placed and delivered to RDMAP. So, I do not think you can say that the message is not placed when the PD ID of the STag to be invalidated and PD ID of RDMAP Stream are not the same. Under RDMAP Stream association, a given RDMAP Stream is identified locally by a unique RDMAP Stream identifier (ID) and that RDMA Stream ID is associated with the STag and RDMAP Stream. For an incoming RDMA Read Request Message on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the Data Source STag, then the Message is not placed and the RDMAP Stream MUST be Terminated with an Invalid STag error. Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated with an STag cannot be Invalidated error. Note that the RDMA Stream ID is locally defined, and cannot be directly manipulated by the Remote Peer. Same comment as the paragraph for PD. In case of SendInv and SendSEInv, when you check for the STag association in RDMAP layer, the Untagged DDP Message has been already placed and delivered to RDMAP. So, I do not think you can say that the message is not placed when the RDMAP Stream ID of the STag to be invalidated and RDMAP Stream ID of RDMAP Stream are not the same. Note: For an incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream, the DDP layer MUST associate the STag targeted by the RDMA Write or RDMA Read Response Message (respectively) to the RDMAP Stream. If an STag targeted by the RDMA Write or RDMA Read Response Message Segment is not associated with the RDMAP Stream that received the Message Segment, DDP MUST surface an Invalid STag error to the RDMAP layer. The RDMAP layer MUST Terminate the RDMAP Stream if the DDP Layer surfaces an Invalid STag error. Note with slight modifications: Note: For an incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream, the DDP layer MUST verify the association of the STag targeted by the RDMA Write or RDMA Read Response Message (respectively) and the underlying DDP Stream that received the Message. If an STag targeted by the RDMA Write or RDMA Read Response Message Segment is not associated with the underlying DDP Stream that received the Message Segment, DDP MUST surface an Invalid STag error to the RDMAP layer. The RDMAP layer MUST Terminate the RDMAP Stream if the DDP Layer surfaces an Invalid STag error. I believe the above text resolves the problem Caitlin surfaced, folks, please let me know if you agree. Also, please note, the STag cannot be Invalidated error is one of the two new error codes I surfaced on this reflector. Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 13 13:56:41 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA22094 for ; Thu, 13 Feb 2003 13:56:40 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1DIxvL10872 for rddp-archive@odin.ietf.org; Thu, 13 Feb 2003 13:59:57 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1DIxup10869 for ; Thu, 13 Feb 2003 13:59:56 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA22081 for ; Thu, 13 Feb 2003 13:56:09 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1DIvOp10820; Thu, 13 Feb 2003 13:57:24 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1DIuDp10781 for ; Thu, 13 Feb 2003 13:56:13 -0500 Received: from msgbas1x.cos.agilent.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA22035 for ; Thu, 13 Feb 2003 13:52:25 -0500 (EST) From: pat_thaler@agilent.com Received: from relcos1.cos.agilent.com (relcos1.cos.agilent.com [130.29.152.239]) by msgbas1x.cos.agilent.com (Postfix) with ESMTP id 6CA9715E07; Thu, 13 Feb 2003 11:56:09 -0700 (MST) Received: from axcsbh1.cos.agilent.com (axcsbh1.cos.agilent.com [130.29.152.143]) by relcos1.cos.agilent.com (Postfix) with SMTP id 3BDB844B; Thu, 13 Feb 2003 11:56:09 -0700 (MST) Received: from 130.29.152.143 by axcsbh1.cos.agilent.com (InterScan E-Mail VirusWall NT); Thu, 13 Feb 2003 11:56:07 -0700 Received: by axcsbh1.cos.agilent.com with Internet Mail Service (5.5.2653.19) id <1RKJJGSW>; Thu, 13 Feb 2003 11:56:06 -0700 Message-ID: <1BEBA5E8600DD4119A50009027AF54A010CA0D19@axcs04.cos.agilent.com> To: recio@us.ibm.com, rddp@ietf.org Subject: RE: [rddp] Association of STag and RDMAP Stream(s) Date: Thu, 13 Feb 2003 11:56:01 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Renato, This part of the text isn't quite right: For an incoming RDMA Read Request Message on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the Data Source STag, the Message is not placed and the RDMAP Stream MUST be Terminated. ^^^^^^^^^^^^^^^^^^^^^^^^^ Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated. ^^^^^^^^^^^^^^^^^^^^^^^^^ For RDMA Read Request, Send with Invalidate and Send with SE and Invalidate, DDP is doing the placement before it delivers the message to RDMAP and RDMAP is doing the STag validity check. The messages will be placed. The underlined text for RDMA Read Request should be replaced with "no RDMA Read Response is sent". The underlined text for Send with Invalidate and Send with SE and Invalidate should be "the Message is not delivered to the ULP". The corrections also apply to the Stream association paragraph. Regards, Pat -----Original Message----- From: Renato Recio [mailto:recio@us.ibm.com] Sent: Tuesday, February 11, 2003 8:06 AM To: rddp@ietf.org Subject: [rddp] Association of STag and RDMAP Stream(s) Folks, A problem was raised on this reflector regarding the need for a little more text describing the mechanisms the RDMAP layer may use to associate an RDMAP Stream and an STag. Following is text I propose to add to bullet #5 in section 10.3 of the -01 specification: RDMAP MAY choose several locally defined mechanisms for associating a RDMAP Stream and a STag. One such mechanism is to provide two association types: a protection domain association and an RDMAP Stream association. Under the protection domain (PD) association, a unique protection domain identifier (PD ID) is created and used locally to associate the STag with a set of RDMAP Streams. Under this mechanism, only RDMAP Streams that have the same PD ID as the STag are allowed to use the STag. For an incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the STag targeted by the RDMA Write or RDMA Read Response Message (respectively), then the Message is not placed and the RDMAP Stream MUST be Terminated. For an incoming RDMA Read Request Message on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the Data Source STag, the Message is not placed and the RDMAP Stream MUST be Terminated. Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated. Note that the PD ID is locally defined, and cannot be directly manipulated by the Remote Peer. Under RDMAP Stream association, a given RDMAP Stream is identified locally by a unique RDMAP Stream identifier (ID) and that RDMA Stream ID is associated with the STag and RDMAP Stream. In this case, for an incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the STag targeted by the RDMA Write or RDMA Read Response Message (respectively), then the Message is not placed and the RDMAP Stream MUST be Terminated. Similarly, for an incoming RDMA Read Request Message on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the Data Source STag, then the Message is not placed and the RDMAP Stream MUST be Terminated. Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated. Note that the RDMA Stream ID is locally defined, and cannot be directly manipulated by the Remote Peer. I believe the above text resolves the problem, folks, please let me know if you agree. Thanks, Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 13 14:12:17 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22449 for ; Thu, 13 Feb 2003 14:12:17 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1DJFYT12353 for rddp-archive@odin.ietf.org; Thu, 13 Feb 2003 14:15:34 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1DJFXp12350 for ; Thu, 13 Feb 2003 14:15:33 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22428 for ; Thu, 13 Feb 2003 14:11:46 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1DJEqp12302; Thu, 13 Feb 2003 14:14:52 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1DJDmp12241 for ; Thu, 13 Feb 2003 14:13:48 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22393 for ; Thu, 13 Feb 2003 14:10:00 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id AC2872448C6; Thu, 13 Feb 2003 14:13:43 -0500 (EST) Date: Thu, 13 Feb 2003 13:14:58 -0600 From: Caitlin Bestler Subject: Re: [rddp] Association of STag and RDMAP Stream(s) - version 3 To: Renato Recio Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/11/03, Renato Recio wrote: > > Under the protection domain (PD) association, a unique ^^^^^^ > protection domain identifier (PD ID) is created and used > locally to associate the STag with a set of DDP Streams. The use of "unique" as an adjective begs the question "over what domain?". I believe there is a danger of implementers falsely assuming they know what that scope is. Isn't it obvious that the scope is [the interface, the adapter, the host, the IP address]? I believe adding a sentence to the effect that "The scope of the protection domain identifier (PD ID) MUST include at least all of the DDP Streams associated with it. The scope MAY be larger, but to what extent is undefined." Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 13 14:28:28 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22867 for ; Thu, 13 Feb 2003 14:28:28 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1DJVjs13172 for rddp-archive@odin.ietf.org; Thu, 13 Feb 2003 14:31:45 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1DJVjp13169 for ; Thu, 13 Feb 2003 14:31:45 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22860 for ; Thu, 13 Feb 2003 14:27:57 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1DJV2p13151; Thu, 13 Feb 2003 14:31:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1DJRRp12891 for ; Thu, 13 Feb 2003 14:27:27 -0500 Received: from ztxmail03.ztx.compaq.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22741 for ; Thu, 13 Feb 2003 14:23:38 -0500 (EST) Received: from cceexg12.americas.cpqcorp.net (cceexg12.americas.cpqcorp.net [16.110.250.124]) by ztxmail03.ztx.compaq.com (Postfix) with ESMTP id 58C464DA2 for ; Thu, 13 Feb 2003 13:27:23 -0600 (CST) Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg12.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Thu, 13 Feb 2003 13:27:23 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: RE: [rddp] Association of STag and RDMAP Stream(s) Date: Thu, 13 Feb 2003 13:27:23 -0600 Message-ID: <4D027986353D1341ADA4F2A4F8A170762975A36C@cceexc18.americas.cpqcorp.net> Thread-Topic: [rddp] Association of STag and RDMAP Stream(s) Thread-Index: AcLTk0owj+Wz4Fh9RkKqrUH2LYp4zwAAgbSQ From: "Culley, Paul" To: X-OriginalArrivalTime: 13 Feb 2003 19:27:23.0190 (UTC) FILETIME=[EEED1D60:01C2D395] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h1DJRRp12892 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit For the Send with invalidate cases, I disagree with Pat, I think the correct action is to inform the consumer that the invalidate failed, and why, (Invalidate Stag PD invalid), and to do this when the message is delivered to the consumer. Assuming there are not other errors, the Send message is placeable and deliverable. Paul R. Culley HP Senior Fellow 281-514-5543 -----Original Message----- From: pat_thaler@agilent.com [mailto:pat_thaler@agilent.com] Sent: Thursday, February 13, 2003 12:56 PM To: recio@us.ibm.com; rddp@ietf.org Subject: RE: [rddp] Association of STag and RDMAP Stream(s) Renato, This part of the text isn't quite right: For an incoming RDMA Read Request Message on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the Data Source STag, the Message is not placed and the RDMAP Stream MUST be Terminated. ^^^^^^^^^^^^^^^^^^^^^^^^^ Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated. ^^^^^^^^^^^^^^^^^^^^^^^^^ For RDMA Read Request, Send with Invalidate and Send with SE and Invalidate, DDP is doing the placement before it delivers the message to RDMAP and RDMAP is doing the STag validity check. The messages will be placed. The underlined text for RDMA Read Request should be replaced with "no RDMA Read Response is sent". The underlined text for Send with Invalidate and Send with SE and Invalidate should be "the Message is not delivered to the ULP". The corrections also apply to the Stream association paragraph. Regards, Pat -----Original Message----- From: Renato Recio [mailto:recio@us.ibm.com] Sent: Tuesday, February 11, 2003 8:06 AM To: rddp@ietf.org Subject: [rddp] Association of STag and RDMAP Stream(s) Folks, A problem was raised on this reflector regarding the need for a little more text describing the mechanisms the RDMAP layer may use to associate an RDMAP Stream and an STag. Following is text I propose to add to bullet #5 in section 10.3 of the -01 specification: RDMAP MAY choose several locally defined mechanisms for associating a RDMAP Stream and a STag. One such mechanism is to provide two association types: a protection domain association and an RDMAP Stream association. Under the protection domain (PD) association, a unique protection domain identifier (PD ID) is created and used locally to associate the STag with a set of RDMAP Streams. Under this mechanism, only RDMAP Streams that have the same PD ID as the STag are allowed to use the STag. For an incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the STag targeted by the RDMA Write or RDMA Read Response Message (respectively), then the Message is not placed and the RDMAP Stream MUST be Terminated. For an incoming RDMA Read Request Message on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the Data Source STag, the Message is not placed and the RDMAP Stream MUST be Terminated. Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated. Note that the PD ID is locally defined, and cannot be directly manipulated by the Remote Peer. Under RDMAP Stream association, a given RDMAP Stream is identified locally by a unique RDMAP Stream identifier (ID) and that RDMA Stream ID is associated with the STag and RDMAP Stream. In this case, for an incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the STag targeted by the RDMA Write or RDMA Read Response Message (respectively), then the Message is not placed and the RDMAP Stream MUST be Terminated. Similarly, for an incoming RDMA Read Request Message on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the Data Source STag, then the Message is not placed and the RDMAP Stream MUST be Terminated. Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated. Note that the RDMA Stream ID is locally defined, and cannot be directly manipulated by the Remote Peer. I believe the above text resolves the problem, folks, please let me know if you agree. Thanks, Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 13 14:56:36 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA23584 for ; Thu, 13 Feb 2003 14:56:36 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1DJxsg15232 for rddp-archive@odin.ietf.org; Thu, 13 Feb 2003 14:59:54 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1DJxrp15229 for ; Thu, 13 Feb 2003 14:59:53 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA23576 for ; Thu, 13 Feb 2003 14:56:04 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1DJx9p15185; Thu, 13 Feb 2003 14:59:09 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1DJsqp14968 for ; Thu, 13 Feb 2003 14:54:52 -0500 Received: from msgbas2x.cos.agilent.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA23441 for ; Thu, 13 Feb 2003 14:51:03 -0500 (EST) From: pat_thaler@agilent.com Received: from relcos2.cos.agilent.com (relcos2.cos.agilent.com [130.29.152.237]) by msgbas2x.cos.agilent.com (Postfix) with ESMTP id A5E2822CD; Thu, 13 Feb 2003 12:54:47 -0700 (MST) Received: from axcsbh4.cos.agilent.com (axcsbh4.cos.agilent.com [130.29.152.145]) by relcos2.cos.agilent.com (Postfix) with SMTP id 727E6546; Thu, 13 Feb 2003 12:54:47 -0700 (MST) Received: from 130.29.152.145 by axcsbh4.cos.agilent.com (InterScan E-Mail VirusWall NT); Thu, 13 Feb 2003 12:54:45 -0700 Received: by axcsbh4.cos.agilent.com with Internet Mail Service (5.5.2653.19) id ; Thu, 13 Feb 2003 12:54:45 -0700 Message-ID: <1BEBA5E8600DD4119A50009027AF54A010CA0D5A@axcs04.cos.agilent.com> To: Paul.Culley@hp.com, rddp@ietf.org Subject: RE: [rddp] Association of STag and RDMAP Stream(s) Date: Thu, 13 Feb 2003 12:54:42 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Paul, I considered that possibility and I don't feel strongly about it. However, what I proposed is consistant with what is done for other error cases. The stream is being terminated. Do you deliver the last errored message or not. In other error cases where we terminate, I don't think we specify delivering the message though we do pass the headers in the terminate message and I assume that an implementation would make at least the header available locally for debug purposes. One knows that either something is broken or someone is maliciously trying to invalidate an STag they shouldn't access. Other then debug, one wouldn't want the ULP to act on the message which might have been malicious. One could deliver the message with an error indication and rely on the ULP to not act on it. It isn't clear that that accomplishes a useful purpose. Regards, Pat -----Original Message----- From: Culley, Paul [mailto:Paul.Culley@hp.com] Sent: Thursday, February 13, 2003 11:27 AM To: rddp@ietf.org Subject: RE: [rddp] Association of STag and RDMAP Stream(s) For the Send with invalidate cases, I disagree with Pat, I think the correct action is to inform the consumer that the invalidate failed, and why, (Invalidate Stag PD invalid), and to do this when the message is delivered to the consumer. Assuming there are not other errors, the Send message is placeable and deliverable. Paul R. Culley HP Senior Fellow 281-514-5543 -----Original Message----- From: pat_thaler@agilent.com [mailto:pat_thaler@agilent.com] Sent: Thursday, February 13, 2003 12:56 PM To: recio@us.ibm.com; rddp@ietf.org Subject: RE: [rddp] Association of STag and RDMAP Stream(s) Renato, This part of the text isn't quite right: For an incoming RDMA Read Request Message on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the Data Source STag, the Message is not placed and the RDMAP Stream MUST be Terminated. ^^^^^^^^^^^^^^^^^^^^^^^^^ Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated. ^^^^^^^^^^^^^^^^^^^^^^^^^ For RDMA Read Request, Send with Invalidate and Send with SE and Invalidate, DDP is doing the placement before it delivers the message to RDMAP and RDMAP is doing the STag validity check. The messages will be placed. The underlined text for RDMA Read Request should be replaced with "no RDMA Read Response is sent". The underlined text for Send with Invalidate and Send with SE and Invalidate should be "the Message is not delivered to the ULP". The corrections also apply to the Stream association paragraph. Regards, Pat -----Original Message----- From: Renato Recio [mailto:recio@us.ibm.com] Sent: Tuesday, February 11, 2003 8:06 AM To: rddp@ietf.org Subject: [rddp] Association of STag and RDMAP Stream(s) Folks, A problem was raised on this reflector regarding the need for a little more text describing the mechanisms the RDMAP layer may use to associate an RDMAP Stream and an STag. Following is text I propose to add to bullet #5 in section 10.3 of the -01 specification: RDMAP MAY choose several locally defined mechanisms for associating a RDMAP Stream and a STag. One such mechanism is to provide two association types: a protection domain association and an RDMAP Stream association. Under the protection domain (PD) association, a unique protection domain identifier (PD ID) is created and used locally to associate the STag with a set of RDMAP Streams. Under this mechanism, only RDMAP Streams that have the same PD ID as the STag are allowed to use the STag. For an incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the STag targeted by the RDMA Write or RDMA Read Response Message (respectively), then the Message is not placed and the RDMAP Stream MUST be Terminated. For an incoming RDMA Read Request Message on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the Data Source STag, the Message is not placed and the RDMAP Stream MUST be Terminated. Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the PD ID associated with that RDMAP Stream is not the same as the PD ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated. Note that the PD ID is locally defined, and cannot be directly manipulated by the Remote Peer. Under RDMAP Stream association, a given RDMAP Stream is identified locally by a unique RDMAP Stream identifier (ID) and that RDMA Stream ID is associated with the STag and RDMAP Stream. In this case, for an incoming RDMA Write or RDMA Read Response Message on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the STag targeted by the RDMA Write or RDMA Read Response Message (respectively), then the Message is not placed and the RDMAP Stream MUST be Terminated. Similarly, for an incoming RDMA Read Request Message on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the Data Source STag, then the Message is not placed and the RDMAP Stream MUST be Terminated. Finally, for a Send with Invalidate or Send with SE and Invalidate on an RDMAP Stream, if the RDMAP Stream ID associated with that RDMAP Stream is not the same as the RDMAP Stream ID associated with the STag that is to be Invalidated, the Message is not placed and the RDMAP Stream MUST be Terminated. Note that the RDMA Stream ID is locally defined, and cannot be directly manipulated by the Remote Peer. I believe the above text resolves the problem, folks, please let me know if you agree. Thanks, Renato J Recio IBM Distinguished Engineer Member IBM Academy of Technology IBM I/O Architecture and Technology Tel 512-838-3685, T/L 678-3685 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 17 15:25:58 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA00500 for ; Mon, 17 Feb 2003 15:25:58 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1HKVEg20330 for rddp-archive@odin.ietf.org; Mon, 17 Feb 2003 15:31:14 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1HKVEp20327 for ; Mon, 17 Feb 2003 15:31:14 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA00483 for ; Mon, 17 Feb 2003 15:25:26 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1HKUGp20258; Mon, 17 Feb 2003 15:30:16 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1HKTrp20222 for ; Mon, 17 Feb 2003 15:29:53 -0500 Received: from mxic2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA00456 for ; Mon, 17 Feb 2003 15:24:05 -0500 (EST) From: Black_David@emc.com Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRHL5X1>; Mon, 17 Feb 2003 15:27:54 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C8C7@corpmx14.us.dg.com> To: recio@us.ibm.com, rddp@ietf.org Subject: RE: [rddp] Association of STag and RDMAP Stream(s) - version 3 Date: Mon, 17 Feb 2003 15:25:42 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This needs a "MUST implement this functionality or equivalent" requirement on implementations, including a requirement for interfaces to enable every ULP to control this behavior of STags that it exports. Other association models could be allowed, but these two would need to be mandatory. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- > -----Original Message----- > From: Renato Recio [mailto:recio@us.ibm.com] > Sent: Tuesday, February 11, 2003 2:02 PM > To: rddp@ietf.org > Subject: [rddp] Association of STag and RDMAP Stream(s) - version 3 > > > > > > > Sorry folks, this is a resend, the second set of text is for the RDMA > spec. > > I believe the following text addresses the issue raised by Caitlin > regarding STag validation. > > Following is text I propose to add to bullet #3 in section 10.3 of the DDP > -01 specification: > > DDP MAY choose several locally defined mechanisms for associating a DDP > Stream and a STag. One such mechanism is to provide two association > types: a protection domain association and an DDP Stream association. > > Under the protection domain (PD) association, a unique protection > domain identifier (PD ID) is created and used locally to associate > the STag with a set of DDP Streams. Under this mechanism, only DDP > Streams that have the same PD ID as the STag are allowed to use the > STag. For an incoming Tagged Message on an DDP Stream, if the PD ID > associated with that DDP Stream is not the same as the PD ID > associated with the STag targeted by the Tagged Message, then the > Message is not placed and the DDP Layer MUST surface a local error to > the ULP. Note that the PD ID is locally defined, and cannot be > directly manipulated by the Remote Peer. > > Under DDP Stream association, a given DDP Stream is identified > locally by a unique DDP Stream identifier (ID) and that DDP Stream ID > is associated with the STag and DDP Stream. In this case, for an > incoming Tagged Message on an DDP Stream, if the DDP Stream ID > associated with that DDP Stream is not the same as the DDP Stream ID > associated with the STag targeted by the Tagged Message, then the > Message is not placed and and the DDP Layer MUST surface a local > error to the ULP. Note that the DDP Stream ID is locally defined, and > cannot be directly manipulated by the Remote Peer. > > > Following is text I propose to add to bullet #5 in section 10.3 of the > RDMAP -01 specification: > > > RDMAP MAY choose several locally defined mechanisms for associating a > RDMAP Stream and a STag. One such mechanism is to provide two > association types: a protection domain association and an RDMAP Stream > association. > > Under the protection domain (PD) association, a unique protection > domain identifier (PD ID) is created and used locally to associate > the STag with a set of RDMAP Streams. Under this mechanism, only > RDMAP Streams that have the same PD ID as the STag are allowed to use > the STag. > > For an incoming RDMA Read Request Message on an RDMAP Stream, if > the PD ID associated with that RDMAP Stream is not the same as the > PD ID associated with the Data Source STag, the Message is not > placed and the RDMAP Stream MUST be Terminated with an Invalid > STag error. Finally, for a Send with Invalidate or Send with SE > and Invalidate on an RDMAP Stream, if the PD ID associated with > that RDMAP Stream is not the same as the PD ID associated with the > STag that is to be Invalidated, the Message is not placed and the > RDMAP Stream MUST be Terminated with an STag cannot be Invalidated > error. Note that the PD ID is locally defined, and cannot be > directly manipulated by the Remote Peer. > > Under RDMAP Stream association, a given RDMAP Stream is identified > locally by a unique RDMAP Stream identifier (ID) and that RDMA Stream > ID is associated with the STag and RDMAP Stream. > > For an incoming RDMA Read Request Message on an RDMAP Stream, if > the RDMAP Stream ID associated with that RDMAP Stream is not the > same as the RDMAP Stream ID associated with the Data Source STag, > then the Message is not placed and the RDMAP Stream MUST be > Terminated with an Invalid STag error. Finally, for a Send with > Invalidate or Send with SE and Invalidate on an RDMAP Stream, if > the RDMAP Stream ID associated with that RDMAP Stream is not the > same as the RDMAP Stream ID associated with the STag that is to be > Invalidated, the Message is not placed and the RDMAP Stream MUST > be Terminated with an STag cannot be Invalidated error. Note that > the RDMA Stream ID is locally defined, and cannot be directly > manipulated by the Remote Peer. > > Note: For an incoming RDMA Write or RDMA Read Response Message on an > RDMAP Stream, the DDP layer MUST associate the STag targeted by the RDMA > Write or RDMA Read Response Message (respectively) to the RDMAP Stream. > If an STag targeted by the RDMA Write or RDMA Read Response Message > Segment is not associated with the RDMAP Stream that received the > Message Segment, DDP MUST surface an Invalid STag error to the RDMAP > layer. The RDMAP layer MUST Terminate the RDMAP Stream if the DDP Layer > surfaces an Invalid STag error. > > I believe the above text resolves the problem Caitlin surfaced, folks, > please let me know if you agree. Also, please note, the STag cannot be > Invalidated error is one of the two new error codes I surfaced on this > reflector. > > Renato J Recio > IBM Distinguished Engineer > Member IBM Academy of Technology > IBM I/O Architecture and Technology > Tel 512-838-3685, T/L 678-3685 > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 17 15:36:30 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA00832 for ; Mon, 17 Feb 2003 15:36:30 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1HKfk721582 for rddp-archive@odin.ietf.org; Mon, 17 Feb 2003 15:41:46 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1HKfkp21579 for ; Mon, 17 Feb 2003 15:41:46 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA00825 for ; Mon, 17 Feb 2003 15:35:59 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1HKf4p21520; Mon, 17 Feb 2003 15:41:04 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1HKdTp21388 for ; Mon, 17 Feb 2003 15:39:29 -0500 Received: from mxic1.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA00764 for ; Mon, 17 Feb 2003 15:33:41 -0500 (EST) From: Black_David@emc.com Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1KDYZX8T>; Mon, 17 Feb 2003 15:37:30 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C8C8@corpmx14.us.dg.com> To: Jim.Williams@emulex.com, rddp@ietf.org Subject: RE: [rddp] STag Invalidation Question Date: Mon, 17 Feb 2003 15:35:20 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , WG Chair hat off ... > pat_thaler wrote: > > > Your write of a file from a client to a server is one example > > of this. Another example is the case where a client ULP > > requests its server ULP to write a file. The client sends an > > STag in the request to advertise the buffer for the file. The > > server may not have all the data ready for the transfer at > > once and the client shouldn't have to know that detail in > > order to set up its buffer. When the server ULP has data > > ready to transfer to the client, it will start an RDMA Write > > for that data. Note that an RDMAP receiving RDMA Writes does > > not deliver them the ULP. Whether the request takes one RDMA > > Write or many is transparent to the client ULP. > > You make a good point. I think I agree with this. > Need to think a bit more about the whole problem. I don't see a major issue with this sort of interface paradigm: - The receiving ULP wants the entire lump, and doesn't care how many pieces it arrives in. The receiving ULP isn't going to do anything until all the data arrives, and can arrange for the STag to be invalid at that point. Caitlin's time-limited delivery scenario is somewhat different: - The receiving ULP definitely wants to do something with the data while leaving the STag valid for more to arrive. One possibility that would allow the receiving ULP to protect itself would be to have some way to change the STag range without affecting its validity (e.g., a Send arrives saying "40K made it through in the last time period before I sent this", so the receiving ULP moves the lower bound of the STag up by 40K before starting to use the data). This would need to be considered vs. advising the ULP to use an alternative design of alternating (or round-robining) among a buffers/STags, where the buffer switch is time-driven (e.g., and a Send indicates how much data was sent) to determine how important it is to be able to change the region of memory described by an STag without invalidating the STag. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 18 11:42:14 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA01590 for ; Tue, 18 Feb 2003 11:42:14 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1IGlr307856 for rddp-archive@odin.ietf.org; Tue, 18 Feb 2003 11:47:53 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1IGlrp07853 for ; Tue, 18 Feb 2003 11:47:53 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA01578 for ; Tue, 18 Feb 2003 11:41:43 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1IGl9p07818; Tue, 18 Feb 2003 11:47:09 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1IGkDp07783 for ; Tue, 18 Feb 2003 11:46:13 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA01511 for ; Tue, 18 Feb 2003 11:40:02 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRK2862>; Tue, 18 Feb 2003 11:43:50 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C8CA@corpmx14.us.dg.com> To: rddp@ietf.org Date: Tue, 18 Feb 2003 11:41:37 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] MPA TCP analysis draft Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This draft (draft-elzur-iwarp-mpa-tcp-analysis-00.txt) has been out for over a week and generated no discussion. Let me see if I can engender some discussion by offering some observations in my role as WG chair. All of these are preliminary observations for further discussion, but I would ask that folks pay particular attention to the "prove" issue in (3) to avoid the WG chair having to get out a clue-by-four. The Elzur MPA TCP analysis draft covers 3 topics: (1) Packing ULPDUs into a TCP segment (2) Sender alignment of ULPDUs to the start of TCP segments (3) Avoiding endpoint confusion about whether MPA is enabled Discussion of each follows: -- (1) Packing The draft recommends that packing of multiple ULPDUs into a single TCP segment be supported (sender MAY pack, receiver MUST support receipt of packed segments). This strikes me as a good thing and a welcome change. -- (2) Sender ULPDU alignment to TCP segments Some of the buffer size calculations seem questionable given the prior WG consensus that senders using unmodified TCP stacks (which cannot support alignment in all circumstances) be supported - a sender running over an unmodified stack will lose alignment despite its best efforts, so the claimed benefits of alignment are probably overstated, BUT - see the end of the discussion of (3) below on identifying sender behavior. OTOH, the calculation of buffering required for a non-aligned MPA sender seems to assume no drops or reordering, although considering those would only increase the advantage of reliable sender alignment. Overall, I tend to agree that there are useful receiver implementation optimizations enabled by a sender who can align ULPDUs to TCP segments. One that I didn't see mentioned is that checking the MPA CRC when each packet is received allows an MPA CRC failure to be dealt with by dropping the offending packet and using TCP retransmission - an approach that is not as readily usable if a CRC spans multiple packets that do not arrive back-to-back. While this is a layer violation (CRC check "above" TCP triggers TCP drop), I would be prepared to argue with the "layering police" that this is acceptable and desirable because it strengthens an existing TCP function (detection and rejection of packets corrupted in transmission), and the pure TCP alternative (TCP Alternate Checksum - RFC 1146) is undeployable in practice due to the changes it requires to both TCP stacks and the wire format (lots of stuff will break). A key design challenge/goal is to ensure that receivers can take advantage of receiver alignment performance optimizations *without* locking out senders that use unmodified TCP stacks due to their inherent ability to lose alignment. -- (3) Avoiding Endpoint Confusion about MPA use I want to remind everyone that our AD asked the WG to "prove" that senders and receivers cannot be confused about whether MPA is in use; I assure everyone our AD chose the word "prove" deliberately. With suitable design attention to the MPA Hello/Hello ACK/ACK details, I believe such a mechanism has a good possibility of providing a suitable "proof" - Figure 5 looks like a good starting point, and its conciseness is a positive indication. OTOH, I have serious concerns about the protocol analysis required (Section 8.2) for transition via a well known port or service location protocol (Figure 4). With apologies to the authors (whose hard work on this draft is acknowledged and appreciated), that analysis reminds me of an old joke about how to "prove" that all odd numbers > 1 are prime: 3 is prime 5 is prime 7 is prime 9 is ... experimental error 11 is prime 13 is prime lacking a counterexample, they must all be prime ... :-) :-) [for "experimental error", see the DNS discussion in Section 8.2.4.3] In all seriousness, the inability to find a counterexample does not prove anything, as there are many more protocols that run over TCP than those considered in Section 8.2, and who knows whether 32 bits of zero is a command code in some private protocol that hasn't been considered. It seems to me that efforts might be better directed towards using an MPA Hello/Hello ACK/ACK or similar mechanism in all cases, even when a well-known port or service location protocol is used for the transition into MPA mode. Such a design would also provide an opportunity to address a nagging buffering issue - a receiver that can't distinguish an MPA-modified aligning TCP sender from an unmodified TCP sender who is nonetheless using MPA (e.g., for the CRC) will have to pessimistically assume that all senders are unmodified TCPs and allocate the larger buffering resources discussed in Section 7.1. In the current MPA design, I don't believe a receiver can make that distinction. In contrast, if different MPA Hello codes were used for the two classes of senders, the receiver could more reliably optimize the buffering for the MPA-modified aligning TCP that should require less buffering. A receiver will still have to watch out for unexpected alignment failures even from MPA-modified aligning TCP senders (e.g., courtesy of Caitlin's resegmenting middleboxes that delay making themselves known). This is all for further discussion ... I would suggest that replies to this message pick out one of the three topics above and change the message subject accordingly. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 18 12:15:04 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02218 for ; Tue, 18 Feb 2003 12:15:04 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1IHKji10433 for rddp-archive@odin.ietf.org; Tue, 18 Feb 2003 12:20:45 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1IHKip10430 for ; Tue, 18 Feb 2003 12:20:44 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02212 for ; Tue, 18 Feb 2003 12:14:33 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1IHK2p10403; Tue, 18 Feb 2003 12:20:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1IHJbp10341 for ; Tue, 18 Feb 2003 12:19:37 -0500 Received: from ztxmail05.ztx.compaq.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02194 for ; Tue, 18 Feb 2003 12:13:25 -0500 (EST) Received: from cceexg13.americas.cpqcorp.net (cceexg13.americas.cpqcorp.net [16.110.250.119]) by ztxmail05.ztx.compaq.com (Postfix) with ESMTP id EF029EFE; Tue, 18 Feb 2003 11:17:12 -0600 (CST) Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg13.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Tue, 18 Feb 2003 11:17:12 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: RE: [rddp] MPA TCP analysis draft; avoiding confusion on use Date: Tue, 18 Feb 2003 11:17:12 -0600 Message-ID: <4D027986353D1341ADA4F2A4F8A170762975A462@cceexc18.americas.cpqcorp.net> Thread-Topic: [rddp] MPA TCP analysis draft; avoiding confusion on use Thread-Index: AcLXbhB4qlEkh4tIQn6wjlll2h5D8QAAfM8Q From: "Culley, Paul" To: , X-OriginalArrivalTime: 18 Feb 2003 17:17:12.0694 (UTC) FILETIME=[9392C160:01C2D771] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h1IHJbp10344 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit I have always been curios why the ADs are placing requirement 3 on MPA. Since MPA is now characterized as an adaptation layer for DDP, and as such, cannot get turned on unless DDP is turned on, why is it necessary to "prove" that MPA/DDP cannot ever get accidentally (or otherwise) connected to an inappropriate port/address and (presumably) cause some kind of trouble? Has this kind of requirement been applied to other applications that use TCP? For example, has it been shown that an HTTP client cannot ever cause a problem when connecting to some other non-http server or other client? I suspect not. Of course it may be that the ADs placed this requirement on MPA when MPA was being put forth as a general framing mechanism; depending on how it got enabled, it might have been enabled unexpectedly under some unsuspecting application. I never thought that was very likely either, but... Attempting to "fix" this "problem" by adding additional Hello exchanges, merely seems to move the problem to "proving" that the Hello messages, cannot cause problems themselves, putting us back where we started... Paul R. Culley HP Senior Fellow 281-514-5543 -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Tuesday, February 18, 2003 10:42 AM To: rddp@ietf.org Subject: [rddp] MPA TCP analysis draft This draft (draft-elzur-iwarp-mpa-tcp-analysis-00.txt) has been out for over a week and generated no discussion. Let me see if I can engender some discussion by offering some observations in my role as WG chair. All of these are preliminary observations for further discussion, but I would ask that folks pay particular attention to the "prove" issue in (3) to avoid the WG chair having to get out a clue-by-four. The Elzur MPA TCP analysis draft covers 3 topics: (1) Packing ULPDUs into a TCP segment (2) Sender alignment of ULPDUs to the start of TCP segments (3) Avoiding endpoint confusion about whether MPA is enabled Discussion of each follows: -- (1) Packing The draft recommends that packing of multiple ULPDUs into a single TCP segment be supported (sender MAY pack, receiver MUST support receipt of packed segments). This strikes me as a good thing and a welcome change. -- (2) Sender ULPDU alignment to TCP segments Some of the buffer size calculations seem questionable given the prior WG consensus that senders using unmodified TCP stacks (which cannot support alignment in all circumstances) be supported - a sender running over an unmodified stack will lose alignment despite its best efforts, so the claimed benefits of alignment are probably overstated, BUT - see the end of the discussion of (3) below on identifying sender behavior. OTOH, the calculation of buffering required for a non-aligned MPA sender seems to assume no drops or reordering, although considering those would only increase the advantage of reliable sender alignment. Overall, I tend to agree that there are useful receiver implementation optimizations enabled by a sender who can align ULPDUs to TCP segments. One that I didn't see mentioned is that checking the MPA CRC when each packet is received allows an MPA CRC failure to be dealt with by dropping the offending packet and using TCP retransmission - an approach that is not as readily usable if a CRC spans multiple packets that do not arrive back-to-back. While this is a layer violation (CRC check "above" TCP triggers TCP drop), I would be prepared to argue with the "layering police" that this is acceptable and desirable because it strengthens an existing TCP function (detection and rejection of packets corrupted in transmission), and the pure TCP alternative (TCP Alternate Checksum - RFC 1146) is undeployable in practice due to the changes it requires to both TCP stacks and the wire format (lots of stuff will break). A key design challenge/goal is to ensure that receivers can take advantage of receiver alignment performance optimizations *without* locking out senders that use unmodified TCP stacks due to their inherent ability to lose alignment. -- (3) Avoiding Endpoint Confusion about MPA use I want to remind everyone that our AD asked the WG to "prove" that senders and receivers cannot be confused about whether MPA is in use; I assure everyone our AD chose the word "prove" deliberately. With suitable design attention to the MPA Hello/Hello ACK/ACK details, I believe such a mechanism has a good possibility of providing a suitable "proof" - Figure 5 looks like a good starting point, and its conciseness is a positive indication. OTOH, I have serious concerns about the protocol analysis required (Section 8.2) for transition via a well known port or service location protocol (Figure 4). With apologies to the authors (whose hard work on this draft is acknowledged and appreciated), that analysis reminds me of an old joke about how to "prove" that all odd numbers > 1 are prime: 3 is prime 5 is prime 7 is prime 9 is ... experimental error 11 is prime 13 is prime lacking a counterexample, they must all be prime ... :-) :-) [for "experimental error", see the DNS discussion in Section 8.2.4.3] In all seriousness, the inability to find a counterexample does not prove anything, as there are many more protocols that run over TCP than those considered in Section 8.2, and who knows whether 32 bits of zero is a command code in some private protocol that hasn't been considered. It seems to me that efforts might be better directed towards using an MPA Hello/Hello ACK/ACK or similar mechanism in all cases, even when a well-known port or service location protocol is used for the transition into MPA mode. Such a design would also provide an opportunity to address a nagging buffering issue - a receiver that can't distinguish an MPA-modified aligning TCP sender from an unmodified TCP sender who is nonetheless using MPA (e.g., for the CRC) will have to pessimistically assume that all senders are unmodified TCPs and allocate the larger buffering resources discussed in Section 7.1. In the current MPA design, I don't believe a receiver can make that distinction. In contrast, if different MPA Hello codes were used for the two classes of senders, the receiver could more reliably optimize the buffering for the MPA-modified aligning TCP that should require less buffering. A receiver will still have to watch out for unexpected alignment failures even from MPA-modified aligning TCP senders (e.g., courtesy of Caitlin's resegmenting middleboxes that delay making themselves known). This is all for further discussion ... I would suggest that replies to this message pick out one of the three topics above and change the message subject accordingly. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 18 12:37:01 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02687 for ; Tue, 18 Feb 2003 12:37:00 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1IHgft12089 for rddp-archive@odin.ietf.org; Tue, 18 Feb 2003 12:42:41 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1IHgfp12086 for ; Tue, 18 Feb 2003 12:42:41 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02663 for ; Tue, 18 Feb 2003 12:36:29 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1IHfxp12034; Tue, 18 Feb 2003 12:41:59 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1IHbAp11400 for ; Tue, 18 Feb 2003 12:37:10 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02533 for ; Tue, 18 Feb 2003 12:30:58 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRKJCPB>; Tue, 18 Feb 2003 12:34:45 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C8CB@corpmx14.us.dg.com> To: Paul.Culley@hp.com, Black_David@emc.com, rddp@ietf.org Subject: RE: [rddp] MPA TCP analysis draft; avoiding confusion on use Date: Tue, 18 Feb 2003 12:32:32 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Paul, > I have always been curios why the ADs are placing requirement 3 on MPA. > Since MPA is now characterized as an adaptation layer for DDP, and as > such, cannot get turned on unless DDP is turned on, why is it necessary > to "prove" that MPA/DDP cannot ever get accidentally (or otherwise) > connected to an inappropriate port/address and (presumably) cause some > kind of trouble? MPA proposes to change TCP - see the sender alignment discussion. If MPA did not propose to change TCP (i.e., all senders could always use unmodified existing stacks) then (portions of) your "applications that use TCP" discussion would apply ... > Has this kind of requirement been applied to other applications that use > TCP? For example, has it been shown that an HTTP client cannot ever > cause a problem when connecting to some other non-http server or other > client? I suspect not. HTTP, etc. require *no* changes to TCP, hence this issue does not arise. > Of course it may be that the ADs placed this requirement on MPA when MPA > was being put forth as a general framing mechanism; depending on how it > got enabled, it might have been enabled unexpectedly under some > unsuspecting application. I never thought that was very likely either, > but... If MPA is now only for DDP's usage, the proposition that DDP is a somehow more important application of TCP that is entitled to make changes to TCP where essentially no other application has ever been permitted to do so is a non-starter, leaving SCTP as the only means of obtaining ULPDU alignment with transmitted segments/chunks. > Attempting to "fix" this "problem" by adding additional Hello exchanges, > merely seems to move the problem to "proving" that the Hello messages, > cannot cause problems themselves, putting us back where we started... Not exactly. The Hello exchanges can be designed to be specific to the connection, etc. At some point one gets to a "proof" without needing absolute certainty - e.g., every security protocol in the world is vulnerable to the (highly improbable) "lucky guess" of some crucial secret key. This is an exercise in making that sort of event sufficiently improbable (and I don't think the levels of assurance that security protocols get to will be required). Compare Figures 4 and 5 in the Elzur draft, and note that at the very least, some careful design will avoid the DNS "near miss". Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 18 16:52:23 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA09469 for ; Tue, 18 Feb 2003 16:52:23 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1ILwAK28893 for rddp-archive@odin.ietf.org; Tue, 18 Feb 2003 16:58:10 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1ILwAp28890 for ; Tue, 18 Feb 2003 16:58:10 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA09464 for ; Tue, 18 Feb 2003 16:51:51 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1ILvNp28824; Tue, 18 Feb 2003 16:57:23 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1ILugp28767 for ; Tue, 18 Feb 2003 16:56:42 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA09377 for ; Tue, 18 Feb 2003 16:50:24 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id CBB12244944 for ; Tue, 18 Feb 2003 16:54:13 -0500 (EST) Date: Tue, 18 Feb 2003 15:55:24 -0600 From: Caitlin Bestler Subject: Re: [rddp] MPA TCP analysis draft To: rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C8CA@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit The MPA analysis draft offers the following formula starting on page 10: > For the case when the FPDU is not aligned with the TCP > segment, it may take, on average, 2 TCP segments to > assemble one FPDU. Therefore, the receiver has to > allocate BufferSizeNAF (Buffer Size, Non-Aligned FPDU) > octets: > > BufferSizeNAF = K1* EMSS * number_of_connections + > K2 * EMSS > > > Where K1 and K2 are implementation dependent constants > and EMSS is the effective maximum segment size. > > For example, a 1 Gbps link with 10,000 connections and an > EMSS of 1500B would require 15 MB of memory. Often the > number of connections used scales with the network speed, > aggravating the situation for higher speeds. The derivation of this formula does not seem to be supported anywhere in the text. In fact I can't figure out where it came from, or what assumptions are being swept under the rug in K1 and K2. As pointed out, 2 adjacent TCP segments should allow the assembly of one FPDU. (In fact it is only the strangest of corner cases where this is not true). So the question becomes, how many TCP segments might have to be buffered before an FPDU is assembled and can be placed? Phrased that way, the answer is obviously half of the TCP segments that can be in flight. The worst case scenario is clearly dropping every other TCP segment. (And as previously noted, if a load-balancing router dispatches successive segments by alternate path that is *exactly* what could happen if one path is delayed). I believe that is a simpler explanation, and formula. It does not change any of the conclusions, however. If the formula presented actually represent a *lower* estimate of the required buffering then the rationale should be explained. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 18 17:16:07 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA10010 for ; Tue, 18 Feb 2003 17:16:07 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1IMLrt31102 for rddp-archive@odin.ietf.org; Tue, 18 Feb 2003 17:21:53 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1IMLrp31099 for ; Tue, 18 Feb 2003 17:21:53 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA09996 for ; Tue, 18 Feb 2003 17:15:36 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1IML3p31055; Tue, 18 Feb 2003 17:21:03 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1IMKrp31032 for ; Tue, 18 Feb 2003 17:20:53 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA09979 for ; Tue, 18 Feb 2003 17:14:36 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 6BEA52450B2; Tue, 18 Feb 2003 17:18:24 -0500 (EST) Date: Tue, 18 Feb 2003 16:19:35 -0600 From: Caitlin Bestler Subject: RE: [rddp] MPA TCP analysis draft; avoiding confusion on use To: rddp@ietf.org Cc: Paul.Culley@hp.com, Black_David@emc.com X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C8CB@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/18/03, Black_David@emc.com wrote: >Paul, > >> I have always been curios why the ADs are placing >> requirement 3 on MPA. Since MPA is now characterized as >> an adaptation layer for DDP, and as such, cannot get >> turned on unless DDP is turned on, why is it necessary >> to "prove" that MPA/DDP cannot ever get accidentally (or >> otherwise) connected to an inappropriate port/address >> and (presumably) cause some kind of trouble? > >MPA proposes to change TCP - see the sender alignment >discussion. If MPA did not propose to change TCP (i.e., >all senders could always use unmodified existing stacks) >then (portions of) your "applications that use TCP" >discussion would apply ... > True, MPA makes some minor changes for an MPA aware sender. However, the resulting stream is fully TCP compliant. Specifically, it is impossible to examine a MPA/TCP stream and *prove* that it could not have been generated by an extremely annoying ULP that just wanted the stream to look like it was an MPA/TCP stream. That leads to two very important points: -- Given the flexibility of base TCP for ad hoc ULPs, there will never be a method to absolutely ensure rapid detection of cross-ULP mismatches for TCP. Any such method would have to alter TCP itself. -- An MPA/TCP stream arriving at the wrong port is no more likely to be mistakenly accepted than any other TCP stream. Indeed, the draft does a heroic job of showing that it is *less* likely. But ultimately I don't see probability as being truly relevant here. Even with reduced probability, the bottom line remains that it is a ULP responsibility to identify the port it will accept service upon, and what services will be accepted on a given port, and to somewhat promptly detect and reject stray connections. Further it should protect itself from damage from arbitrary peers who have not been authenticated. Given that there is no special exposure of the Data Sink's memory until *after* an STag has been exchanged this certainly doesn't qualify as a security hole. I agree with Paul, there is no justification for this to be documented. The message needs to be clear that no ULP is going to be dragooned into using DDP or MPA. Hence the normal procedures for matching ULPs and ports are fully applicable, as complete and/or as flawed as they are for all other IP based ULPs. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 18 18:01:23 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA10713 for ; Tue, 18 Feb 2003 18:01:23 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1IN7AA01699 for rddp-archive@odin.ietf.org; Tue, 18 Feb 2003 18:07:10 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1IN7Ap01696 for ; Tue, 18 Feb 2003 18:07:10 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA10694 for ; Tue, 18 Feb 2003 18:00:52 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1IN6Ap01322; Tue, 18 Feb 2003 18:06:10 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1IN48p01196 for ; Tue, 18 Feb 2003 18:04:08 -0500 Received: from palrel11.hp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA10642 for ; Tue, 18 Feb 2003 17:57:43 -0500 (EST) Received: from psalpha0.cup.hp.com (psalpha0.cup.hp.com [15.13.189.230]) by palrel11.hp.com (Postfix) with ESMTP id 13A1F1C017BE for ; Tue, 18 Feb 2003 15:01:31 -0800 (PST) Received: from mk731915.cup.hp.com (mk731915.cup.hp.com [15.8.80.111]) by psalpha0.cup.hp.com (8.9.3/8.8.6) with ESMTP id PAA19158 for ; Tue, 18 Feb 2003 15:00:09 -0800 (PST) Message-Id: <5.1.0.14.2.20030218145223.0411ac88@esmail.cup.hp.com> X-Sender: krause@esmail.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Tue, 18 Feb 2003 14:52:39 -0800 To: rddp@ietf.org From: Michael Krause Subject: RE: [rddp] MPA TCP analysis draft; avoiding confusion on use In-Reply-To: References: <277DD60FB639D511AC0400B0D068B71E0564C8CB@corpmx14.us.dg.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 04:19 PM 2/18/2003 -0600, you wrote: >On 2/18/03, Black_David@emc.com wrote: > > >Paul, > > > >> I have always been curios why the ADs are placing > >> requirement 3 on MPA. Since MPA is now characterized as > >> an adaptation layer for DDP, and as such, cannot get > >> turned on unless DDP is turned on, why is it necessary > >> to "prove" that MPA/DDP cannot ever get accidentally (or > >> otherwise) connected to an inappropriate port/address > >> and (presumably) cause some kind of trouble? > > > >MPA proposes to change TCP - see the sender alignment > >discussion. If MPA did not propose to change TCP (i.e., > >all senders could always use unmodified existing stacks) > >then (portions of) your "applications that use TCP" > >discussion would apply ... > > >True, MPA makes some minor changes for an MPA aware sender. >However, the resulting stream is fully TCP compliant. > >Specifically, it is impossible to examine a MPA/TCP stream >and *prove* that it could not have been generated by an >extremely annoying ULP that just wanted the stream to >look like it was an MPA/TCP stream. > >That leads to two very important points: > >-- Given the flexibility of base TCP for ad hoc ULPs, there >will never be a method to absolutely ensure rapid detection >of cross-ULP mismatches for TCP. Any such method would have >to alter TCP itself. > >-- An MPA/TCP stream arriving at the wrong port is no more >likely to be mistakenly accepted than any other TCP stream. >Indeed, the draft does a heroic job of showing that it is >*less* likely. > >But ultimately I don't see probability as being truly >relevant here. Even with reduced probability, the bottom >line remains that it is a ULP responsibility to identify >the port it will accept service upon, and what services >will be accepted on a given port, and to somewhat promptly >detect and reject stray connections. Further it should >protect itself from damage from arbitrary peers who have not >been authenticated. > >Given that there is no special exposure of the Data Sink's >memory until *after* an STag has been exchanged this >certainly doesn't qualify as a security hole. > >I agree with Paul, there is no justification for this to be >documented. The message needs to be clear that no ULP is >going to be dragooned into using DDP or MPA. Hence the >normal procedures for matching ULPs and ports are fully >applicable, as complete and/or as flawed as they are for >all other IP based ULPs. Strongly agree with the above. Mike _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 18 18:28:53 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA11384 for ; Tue, 18 Feb 2003 18:28:53 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1INYfj03314 for rddp-archive@odin.ietf.org; Tue, 18 Feb 2003 18:34:41 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1INYfp03311 for ; Tue, 18 Feb 2003 18:34:41 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA11377 for ; Tue, 18 Feb 2003 18:28:22 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1INY1p03289; Tue, 18 Feb 2003 18:34:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1INX5p03258 for ; Tue, 18 Feb 2003 18:33:05 -0500 Received: from mxic1.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA11364 for ; Tue, 18 Feb 2003 18:26:47 -0500 (EST) From: Black_David@emc.com Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1KDY6VXQ>; Tue, 18 Feb 2003 18:30:35 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C8DC@corpmx14.us.dg.com> To: cait@asomi.com, rddp@ietf.org Subject: RE: [rddp] MPA TCP analysis draft; avoiding confusion on use Date: Tue, 18 Feb 2003 18:28:16 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > >> I have always been curios why the ADs are placing > >> requirement 3 on MPA. Since MPA is now characterized as > >> an adaptation layer for DDP, and as such, cannot get > >> turned on unless DDP is turned on, why is it necessary > >> to "prove" that MPA/DDP cannot ever get accidentally (or > >> otherwise) connected to an inappropriate port/address > >> and (presumably) cause some kind of trouble? > > > >MPA proposes to change TCP - see the sender alignment > >discussion. If MPA did not propose to change TCP (i.e., > >all senders could always use unmodified existing stacks) > >then (portions of) your "applications that use TCP" > >discussion would apply ... > > > True, MPA makes some minor changes for an MPA aware sender. > However, the resulting stream is fully TCP compliant. > > Specifically, it is impossible to examine a MPA/TCP stream > and *prove* that it could not have been generated by an > extremely annoying ULP that just wanted the stream to > look like it was an MPA/TCP stream. Unfortunately, that's called "missing the point." A lot of time was spent in Atlanta and afterwards on the topic of TCP being more than just a specification of the byte stream on the wire - surely we don't need to re-cover that ground ... > -- Given the flexibility of base TCP for ad hoc ULPs, there > will never be a method to absolutely ensure rapid detection > of cross-ULP mismatches for TCP. Any such method would have > to alter TCP itself. As I said in an earlier post, "absolutely" is a red herring. OTOH, the WG is welcome to take this position - the likely result will be the removal of sender alignment from MPA. > -- An MPA/TCP stream arriving at the wrong port is no more > likely to be mistakenly accepted than any other TCP stream. > Indeed, the draft does a heroic job of showing that it is > *less* likely. [... snip ...] > I agree with Paul, there is no justification for this to be > documented. The message needs to be clear that no ULP is > going to be dragooned into using DDP or MPA. Hence the > normal procedures for matching ULPs and ports are fully > applicable, as complete and/or as flawed as they are for > all other IP based ULPs. And if MPA were not proposing to change TCP, that would be a reasonable end to this discussion. If DDP/MPA is positioned as just another application that uses TCP (like HTTP, FTP, etc.), tsvwg's response to a request to change TCP will be to enquire what part of the word "NO" has not been understood. If people are serious about wanting to take this approach, sender alignment will probably have to be removed from MPA, at which point the "prove" requirement about avoiding confusion should no longer apply. Launching and losing another argument on this topic will not impress the Transport Area Directors when it comes time to decide whether the RDDP WG will be allowed to continue working after the March IETF meeting ... (yes, rddp has a rechartering event coming up fast - check the WG charter on the IETF web site). Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 18 19:04:26 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA12270 for ; Tue, 18 Feb 2003 19:04:26 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1J0AFm06460 for rddp-archive@odin.ietf.org; Tue, 18 Feb 2003 19:10:15 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1J0AFp06457 for ; Tue, 18 Feb 2003 19:10:15 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA12264 for ; Tue, 18 Feb 2003 19:03:55 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1J09Rp06162; Tue, 18 Feb 2003 19:09:28 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1J089p06008 for ; Tue, 18 Feb 2003 19:08:09 -0500 Received: from caduceus.fm.intel.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA12042 for ; Tue, 18 Feb 2003 19:01:48 -0500 (EST) Received: from petasus.fm.intel.com (petasus.fm.intel.com [10.1.192.37]) by caduceus.fm.intel.com (8.11.6/8.11.6/d: outer.mc,v 1.51 2002/09/23 20:43:23 dmccart Exp $) with ESMTP id h1INxTp09668 for ; Tue, 18 Feb 2003 23:59:29 GMT Received: from fmsmsxvs042.fm.intel.com (fmsmsxvs042.fm.intel.com [132.233.42.128]) by petasus.fm.intel.com (8.11.6/8.11.6/d: inner.mc,v 1.28 2003/01/13 19:44:39 dmccart Exp $) with SMTP id h1J00EI23508 for ; Wed, 19 Feb 2003 00:00:14 GMT Received: from FMSMSX018.fm.intel.com ([132.233.42.197]) by fmsmsxvs042.fm.intel.com (NAVGW 2.5.2.11) with SMTP id M2003021816075732420 ; Tue, 18 Feb 2003 16:07:57 -0800 Received: by fmsmsx018.fm.intel.com with Internet Mail Service (5.5.2653.19) id <1GFTXHN1>; Tue, 18 Feb 2003 16:05:32 -0800 Message-ID: From: "Larsen, Roy K" To: "'Black_David@emc.com'" , cait@asomi.com, rddp@ietf.org Subject: RE: [rddp] MPA TCP analysis draft; avoiding confusion on use Date: Tue, 18 Feb 2003 16:05:31 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="ISO-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Tuesday, February 18, 2003 3:28 PM > To: cait@asomi.com; rddp@ietf.org > Subject: RE: [rddp] MPA TCP analysis draft; avoiding confusion on use > > > > >> I have always been curios why the ADs are placing > > >> requirement 3 on MPA. Since MPA is now characterized as > > >> an adaptation layer for DDP, and as such, cannot get > > >> turned on unless DDP is turned on, why is it necessary > > >> to "prove" that MPA/DDP cannot ever get accidentally (or > > >> otherwise) connected to an inappropriate port/address > > >> and (presumably) cause some kind of trouble? > > > > > >MPA proposes to change TCP - see the sender alignment > > >discussion. If MPA did not propose to change TCP (i.e., > > >all senders could always use unmodified existing stacks) > > >then (portions of) your "applications that use TCP" > > >discussion would apply ... > > > > > True, MPA makes some minor changes for an MPA aware sender. > > However, the resulting stream is fully TCP compliant. > > > > Specifically, it is impossible to examine a MPA/TCP stream > > and *prove* that it could not have been generated by an > > extremely annoying ULP that just wanted the stream to > > look like it was an MPA/TCP stream. > > Unfortunately, that's called "missing the point." A lot of > time was spent in Atlanta and afterwards on the topic of > TCP being more than just a specification of the byte stream > on the wire - surely we don't need to re-cover that ground ... > David, At the risk of missing the point yet again, I have looked at the MPA draft and could not find a requirement that TCP must change to support MPA. It clearly differentiates performance benefits that can be derived with a "MPA aware" TCP implementation but allows interoperable operation with an unmodified TCP stack. As Catlin and others have pointed out above, it is not possible for a receiver to detect whether I am sending from a MPA aware implementation or not. I was not at the Atlanta meeting. Was there a fundamental change to the MPA draft that has not been published yet? Roy > > -- Given the flexibility of base TCP for ad hoc ULPs, there > > will never be a method to absolutely ensure rapid detection > > of cross-ULP mismatches for TCP. Any such method would have > > to alter TCP itself. > > As I said in an earlier post, "absolutely" is a red herring. > OTOH, the WG is welcome to take this position - the likely > result will be the removal of sender alignment from MPA. > > > -- An MPA/TCP stream arriving at the wrong port is no more > > likely to be mistakenly accepted than any other TCP stream. > > Indeed, the draft does a heroic job of showing that it is > > *less* likely. > > [... snip ...] > > > I agree with Paul, there is no justification for this to be > > documented. The message needs to be clear that no ULP is > > going to be dragooned into using DDP or MPA. Hence the > > normal procedures for matching ULPs and ports are fully > > applicable, as complete and/or as flawed as they are for > > all other IP based ULPs. > > And if MPA were not proposing to change TCP, that would be a > reasonable end to this discussion. If DDP/MPA is positioned > as just another application that uses TCP (like HTTP, FTP, > etc.), tsvwg's response to a request to change TCP will be > to enquire what part of the word "NO" has not been understood. > If people are serious about wanting to take this approach, > sender alignment will probably have to be removed from MPA, > at which point the "prove" requirement about avoiding confusion > should no longer apply. > > Launching and losing another argument on this topic will not > impress the Transport Area Directors when it comes time to decide > whether the RDDP WG will be allowed to continue working after > the March IETF meeting ... (yes, rddp has a rechartering event > coming up fast - check the WG charter on the IETF web site). > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > black_david@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 18 19:43:34 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA12912 for ; Tue, 18 Feb 2003 19:43:34 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1J0nOs08832 for rddp-archive@odin.ietf.org; Tue, 18 Feb 2003 19:49:24 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1J0nNp08829 for ; Tue, 18 Feb 2003 19:49:23 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA12885 for ; Tue, 18 Feb 2003 19:43:03 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1J0m6p08807; Tue, 18 Feb 2003 19:48:06 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1J0lBp08786 for ; Tue, 18 Feb 2003 19:47:11 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA12851 for ; Tue, 18 Feb 2003 19:40:49 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRKKH1F>; Tue, 18 Feb 2003 19:44:35 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C8DE@corpmx14.us.dg.com> To: roy.k.larsen@intel.com, rddp@ietf.org Subject: RE: [rddp] MPA TCP analysis draft; avoiding confusion on use Date: Tue, 18 Feb 2003 19:42:22 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Roy, > At the risk of missing the point yet again, I have looked at the MPA draft > and could not find a requirement that TCP must change to support MPA. It > clearly differentiates performance benefits that can be derived with a "MPA > aware" TCP implementation but allows interoperable operation with an > unmodified TCP stack. As Caitlin and others have pointed out above, it is > not possible for a receiver to detect whether I am sending from a MPA aware > implementation or not. From draft-culley-iwarp-mpa-01.txt: 5.1.1 TCP Transmit side To provide optimum performance, an MPA-aware transmit side TCP implementation SHOULD be enabled to: * With an EMSS large enough to contain the FPDU(s), segment the outgoing TCP stream such that the first octet of every TCP Segment begins with an FPDU. That "SHOULD" is a change to TCP. Among other things, it is at odds with the Bulk Transfer discussion in Section 4.2.3.11 of RFC 1122, which I have pointed out before. Please have the respect to avoid spend list bandwidth splitting hairs on whether words like "SHOULD" were/are/could be upper or lower case (either in MPA or RFC 1122), or on discussion of circumstances in which it's ok to disregard a "SHOULD" or a "should". It's very clear to all concerned that MPA intends to change TCP sender segmentation behavior to enable receiver optimizations. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 18 20:20:28 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA14114 for ; Tue, 18 Feb 2003 20:20:28 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1J1QJq10903 for rddp-archive@odin.ietf.org; Tue, 18 Feb 2003 20:26:19 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1J1QJp10900 for ; Tue, 18 Feb 2003 20:26:19 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA14103 for ; Tue, 18 Feb 2003 20:19:57 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1J1PMp10871; Tue, 18 Feb 2003 20:25:22 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1J1Opp10820 for ; Tue, 18 Feb 2003 20:24:51 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id UAA14068 for ; Tue, 18 Feb 2003 20:18:30 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id CEC762448A0; Tue, 18 Feb 2003 20:22:19 -0500 (EST) Date: Tue, 18 Feb 2003 19:23:30 -0600 From: Caitlin Bestler Subject: RE: [rddp] MPA TCP analysis draft; avoiding confusion on use To: Black_David@emc.com, rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C8DE@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/18/03, Black_David@emc.com wrote: >Roy, > >> At the risk of missing the point yet again, I have >> looked at the MPA draft and could not find a requirement >> that TCP must change to support MPA. It clearly >> differentiates performance benefits that can be derived >> with a "MPA aware" TCP implementation but allows >> interoperable operation with an unmodified TCP stack. >> As Caitlin and others have pointed out above, it is not >> possible for a receiver to detect whether I am sending >> from a MPA aware implementation or not. > >From draft-culley-iwarp-mpa-01.txt: > >5.1.1 TCP Transmit side > > To provide optimum performance, an MPA-aware transmit > side TCP implementation SHOULD be enabled to: > > * With an EMSS large enough to contain the FPDU(s), > segment the outgoing TCP stream such that the first octet > of every TCP Segment begins with an FPDU. > >That "SHOULD" is a change to TCP. Among other things, it >is at odds with the Bulk Transfer discussion in Section >4.2.3.11 of RFC 1122, which I have pointed out before. > I would submit that the more relevant section of RFC 1122 is 4.2.4.3: >4.2.4.3 Flush Call > > Some TCP implementations have included a FLUSH call, > which will empty the TCP send queue of any data for which > the user has issued SEND calls but which is still to the > right of the current send window. That is, it flushes as > much queued send data as possible without losing sequence > number synchronization. This is useful for implementing > the "abort output" function of Telnet. MPA 5.1.1's SHOULD can be implemented using a 4.2.4.3 compliant flush call. Therefore it does contradict RFC 1122. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Feb 19 09:24:54 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA25746 for ; Wed, 19 Feb 2003 09:24:54 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1JEV1m13751 for rddp-archive@odin.ietf.org; Wed, 19 Feb 2003 09:31:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JEV1p13742 for ; Wed, 19 Feb 2003 09:31:01 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA25721 for ; Wed, 19 Feb 2003 09:24:22 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JEUIp13665; Wed, 19 Feb 2003 09:30:18 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JERrp13521 for ; Wed, 19 Feb 2003 09:27:53 -0500 Received: from frantic.weston.bsdi.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA25636 for ; Wed, 19 Feb 2003 09:21:14 -0500 (EST) Received: (from dab@localhost) by frantic.weston.bsdi.com (8.11.6/8.10.1) id h1JEOSB26695; Wed, 19 Feb 2003 08:24:28 -0600 (CST) Date: Wed, 19 Feb 2003 08:24:28 -0600 (CST) From: David Borman Message-Id: <200302191424.h1JEOSB26695@frantic.weston.bsdi.com> To: Black_David@emc.com, cait@asomi.com, rddp@ietf.org Subject: RE: [rddp] MPA TCP analysis draft; avoiding confusion on use Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > Date: Tue, 18 Feb 2003 19:23:30 -0600 > From: Caitlin Bestler > Subject: RE: [rddp] MPA TCP analysis draft; avoiding confusion on use > > On 2/18/03, Black_David@emc.com wrote: ... > >From draft-culley-iwarp-mpa-01.txt: > > > >5.1.1 TCP Transmit side > > > > To provide optimum performance, an MPA-aware transmit > > side TCP implementation SHOULD be enabled to: > > > > * With an EMSS large enough to contain the FPDU(s), > > segment the outgoing TCP stream such that the first octet > > of every TCP Segment begins with an FPDU. > > > >That "SHOULD" is a change to TCP. Among other things, it > >is at odds with the Bulk Transfer discussion in Section > >4.2.3.11 of RFC 1122, which I have pointed out before. > > > > I would submit that the more relevant section of RFC 1122 > is 4.2.4.3: > > >4.2.4.3 Flush Call > > > > Some TCP implementations have included a FLUSH call, > > which will empty the TCP send queue of any data for which > > the user has issued SEND calls but which is still to the > > right of the current send window. That is, it flushes as > > much queued send data as possible without losing sequence > > number synchronization. This is useful for implementing > > the "abort output" function of Telnet. > > > MPA 5.1.1's SHOULD can be implemented using a 4.2.4.3 > compliant flush call. Therefore it does contradict RFC 1122. Perhaps you meant "does not contradict"? But no matter, because the FLUSH call *throws away* any unsent data (the data to the right of the send window). I don't think that is what you want. You can implement sender alignment over an unmodified TCP stack. Unfortunately, the most portable way to do that is to have application level per-segment acks, and only allow one outstanding segment at a time. Thus, it'd probably be a performance hit. (Another implementation could be done if you could query the TCP stack to see how much data is waiting to be ACKed, and just wait for that to go to zero before writing the next segment.) -David Borman _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Feb 19 11:02:37 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA29785 for ; Wed, 19 Feb 2003 11:02:36 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1JG8ks22066 for rddp-archive@odin.ietf.org; Wed, 19 Feb 2003 11:08:46 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JG8kp22063 for ; Wed, 19 Feb 2003 11:08:46 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA29776 for ; Wed, 19 Feb 2003 11:02:05 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JG7Dp21547; Wed, 19 Feb 2003 11:07:13 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JG6Rp21157 for ; Wed, 19 Feb 2003 11:06:27 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA29689 for ; Wed, 19 Feb 2003 10:59:47 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 7D58D24548B for ; Wed, 19 Feb 2003 11:03:37 -0500 (EST) Date: Wed, 19 Feb 2003 10:04:50 -0600 From: Caitlin Bestler Subject: RE: [rddp] MPA TCP analysis draft; avoiding confusion on use To: rddp@ietf.org X-Priority: 3 In-Reply-To: <200302191424.h1JEOSB26695@frantic.weston.bsdi.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/19/03, David Borman wrote: >> >> >> MPA 5.1.1's SHOULD can be implemented using a 4.2.4.3 >> compliant flush call. Therefore it does contradict RFC 1122. > >Perhaps you meant "does not contradict"? But no matter, >because the FLUSH call *throws away* any unsent data (the >data to the right of the send window). I don't think that >is what you want. > >You can implement sender alignment over an unmodified TCP >stack. Unfortunately, the most portable way to do that is >to have application level per-segment acks, and only allow >one outstanding segment at a time. Thus, it'd probably be >a performance hit. (Another implementation could be done >if you could query the TCP stack to see how much data is >waiting to be ACKed, and just wait for that to go to zero >before writing the next segment.) > Oops, up too late yesterday. The section I should have cited was >4.2.2.2 Use of Push: RFC-793 Section 2.8 > When an application issues a series of SEND calls without > setting the PUSH flag, the TCP MAY aggregate the data > internally without sending it. Similarly, when a series > of segments is received without the PSH bit, a TCP MAY > queue the data internally without passing it to the > receiving application. > > The PSH bit is not a record marker and is independent of > segment boundaries. The transmitter SHOULD collapse > successive PSH bits when it packetizes data, to send the > largest possible segment. > > A TCP MAY implement PUSH flags on SEND calls. If PUSH > flags are not implemented, then the sending TCP: (1) must > not buffer data indefinitely, and (2) MUST set the PSH > bit in the last buffered segment (i.e., when there is no > more queued data to be sent). > > The discussion in RFC-793 on pages 48, 50, and 74 > erroneously implies that a received PSH flag must be > passed to the application layer. Passing a received PSH > flag to the application layer is now OPTIONAL . > > An application program is logically required to set the > PUSH flag in a SEND call whenever it needs to force > delivery of the data to avoid a communication deadlock. > However, a TCP SHOULD send a maximum-sized segment > whenever possible, to improve performance (see Section > 4.2.3.4 ). Note that it does not send TCP MUST send a maximum-sized segment whenever possible to improve performance. MPA/TCP overrides that recomendation *for the purpose of improving performance*. In my mind this is precisely the type of justification that is required to justify overriding a SHOULD. An argument could be made that the use of this facility is not what was intended when the RFC was drafted. But by that criteria a vast majority of the Web is RFC non-compliant. The criteria has always been interoperability. MPA/TCP does not do anything to a TCP stream that any ULP has not been free to do all along. Therefore it does not change TCP. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Feb 19 12:34:04 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02467 for ; Wed, 19 Feb 2003 12:34:04 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1JHeET29189 for rddp-archive@odin.ietf.org; Wed, 19 Feb 2003 12:40:14 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JHeEp29186 for ; Wed, 19 Feb 2003 12:40:14 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02463 for ; Wed, 19 Feb 2003 12:33:33 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JHdJp29116; Wed, 19 Feb 2003 12:39:19 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JHc5p29074 for ; Wed, 19 Feb 2003 12:38:05 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02362 for ; Wed, 19 Feb 2003 12:31:25 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 067D4244A80 for ; Wed, 19 Feb 2003 12:35:13 -0500 (EST) Date: Wed, 19 Feb 2003 11:36:26 -0600 From: Caitlin Bestler To: rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C8CA@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] MPA Packing Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/18/03, Black_David@emc.com wrote: >This draft (draft-elzur-iwarp-mpa-tcp-analysis-00.txt) has >been out for over a week and generated no discussion. Let >me see if I can engender some discussion by offering some >observations in my role as WG chair. All of these are >preliminary observations for further discussion, but I >would ask that folks pay particular attention to the >"prove" issue in (3) to avoid the WG chair having to get >out a clue-by-four. > >The Elzur MPA TCP analysis draft covers 3 topics: >(1) Packing ULPDUs into a TCP segment >(2) Sender alignment of ULPDUs to the start of TCP segments >(3) Avoiding endpoint confusion about whether MPA is enabled >Discussion of each follows: > >-- (1) Packing > >The draft recommends that packing of multiple ULPDUs into a >single TCP segment be supported (sender MAY pack, receiver >MUST support receipt of packed segments). This strikes me >as a good thing and a welcome change. > This nearly makes each ULPDU the equivalent of an SCTP Data Chunk. Aside from suggesting that the feature be renamed "bundling", to match SCTP, I believe we can take the lead from SCTP to identify relevant issues. SCTP allows the ULP to request that a Data Chunk not be delayed for purposes of bundling, The DDP/SCTP draft specifies invoking that option for untagged messages. A similar requirement/recomendation is probably appropriate for SCTP. SCTP allows re-bundling on retransmission, and explicitly states that the DO_NOT_BUNDLE option is to be ignored on retransmission. While SCTP's rationale is fully applicable to MPA/TCP, there is a major difference. Retry may be implemented entirely at the TCP layer. Indeed it may be handled by a middle-box unaware of MPA. More importantly, the data bytes for a TCP stream should be immutable, not merely equivalent on a retransmit. I believe the draft should be explicit that MPA presents segments to TCP *once* and that retransmission MUST NOT result in re-bundling/re-packing. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Feb 19 12:50:01 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA03013 for ; Wed, 19 Feb 2003 12:50:01 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1JHuBQ30247 for rddp-archive@odin.ietf.org; Wed, 19 Feb 2003 12:56:11 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JHuBp30244 for ; Wed, 19 Feb 2003 12:56:11 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02972 for ; Wed, 19 Feb 2003 12:49:29 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JHt4p30149; Wed, 19 Feb 2003 12:55:04 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JHsCp30092 for ; Wed, 19 Feb 2003 12:54:12 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA02911 for ; Wed, 19 Feb 2003 12:47:30 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRKMPCH>; Wed, 19 Feb 2003 12:51:18 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C8E6@corpmx14.us.dg.com> To: rddp@ietf.org Date: Wed, 19 Feb 2003 12:48:55 -0500 Importance: high X-Priority: 1 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] Reality Check: Avoiding confusion on MPA TCP use Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , I need to be more careful about what I wish for - I forgot how enthusiastic some folks can be about discussing TCP framing and the IETF process issues that surround it (a momentary lapse - it won't happen again, at least not anytime soon ;-) ). This message is an attempt to sort out some of issues raised recently on the list and/or in private and suggest a productive path forward. Let me start with an attempt at a consensus call: I believe the rough consensus of the RDDP WG is to pursue an RDDP over TCP solution that allows cooperating TCP implementations to employ sender alignment, provided that: - Alignment is optional - Receiver support/usage of alignment does not lock out sender implementations that use unmodified TCP stacks. I hope this is not controversial, but send objections to the list (as is usual for any attempted consensus call). In case anyone finds the above to be at odds with some of what I've posted recently, all of my comments along the lines of "doing would require removal of sender alignment from MPA" have been with the intent of identifying unproductive courses of action (i.e., I know the MPA advocates don't want to remove alignment from MPA, hence the intended inference from a statement like "doing would require removal of alignment" is: "doing would be a bad idea"). Assuming the acceptance of the above attempt at consensus, the following three items are *facts*, and should not be discussed further on this list: (1) Applications that use TCP don't make changes to TCP. Sender alignment is a TCP change, so MPA is not just an application. (2) The "proof" requirement is linked to the TCP sender alignment change, so the above rough consensus entails providing a "proof". (3) The Transport ADs hold all the cards, specifically the RDDP WG charter is at high risk of being withdrawn or modified to exclude TCP if the WG insists on extensive exploration of the tarpits of whether TCP is being changed and whether "proof" is necessary. Those with questions/concerns/objections to the above are welcome to contact me directly off the list, but the above three items are not open to change and should not be discussed further on the list (yes, the ADs are watching ...). Turning to a more positive topic, there is a path forward available in the Elzur MPA TCP analysis draft. Section 8.1 says: The third model is by negotiation of MPA transition by the ULP using octet stream messages to accomplish a ULP specific , , three way exchange. If this exchange were made MPA-generic instead of ULP-specific and applied to all transitions to MPA, including those that occur immediately at the start of a connection, then Figure 5 in that draft contains much of the "proof" structure and content. This approach would also provide a means of dealing with the current unrealistic buffer size analysis in Section 7. The problem is that *if*, as a number of people have posted, a receiver cannot tell the difference between the bytestream arriving from an unmodified TCP vs. an MPA-enabled TCP, then a receiver has to assume the former, and cannot achieve the desired "BufferSizeAF = K2 * EMSS". FWIW, the approach of requiring that unmodified TCPs and MPA-enabled TCPs contact receivers on different ports is unlikely to be acceptable, and kicking transport buffer sizing issues up to ULPs to deal with is silly. In addition, this sort of exchange would also provide a means of making the current 512-byte fixed marker interval variable, which is probably going to be needed to seriously deal with unmodified TCP stacks (e.g., are 16 syscalls or other send events/descriptors really reasonable to send an 8k DDP data block?). If I can obtain rough consensus to proceed in this direction (initial MPA-generic Hello/Hello ACK/ACK exchange used in all cases), I can then consult with the ADs on the strength of "proof" that the design of the MPA Hello/Hello ACK/ACK needs to provide. At the moment, I don't understand the level of emotion surrounding the opposition to some sort of initial negotiation. I was under the impression that RDDP connections were likely to be long-lived "elephants" rather than short-lived "mice" and that the networks of primary interest were low-latency (e.g., data center). If that's the case, the latency of the extra round trip and overhead of the extra bytes transferred are going to vanish into the noise (of the three messages, only MPA Hello can't carry data) - what have I missed? As indicated above, any discussion of items (1) - (3) should be sent to me directly off the list. Discussion of MPA-generic negotiation is welcome, but objections to it should suggest alternate means of providing the "proof" and providing receivers with information about sender alignment behavior. I really am trying to move this forward, and I believe sender alignment can be part of the solution, but we need to play the hand we've been dealt, rather than the one we'd like to have. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Feb 19 12:56:00 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA03151 for ; Wed, 19 Feb 2003 12:56:00 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1JI2Av30471 for rddp-archive@odin.ietf.org; Wed, 19 Feb 2003 13:02:10 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JI2Ap30468 for ; Wed, 19 Feb 2003 13:02:10 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA03141 for ; Wed, 19 Feb 2003 12:55:28 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JI12p30429; Wed, 19 Feb 2003 13:01:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JI0rp30404 for ; Wed, 19 Feb 2003 13:00:53 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA03098 for ; Wed, 19 Feb 2003 12:54:10 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRKMPSW>; Wed, 19 Feb 2003 12:57:58 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C8E7@corpmx14.us.dg.com> To: cait@asomi.com, rddp@ietf.org Subject: RE: [rddp] MPA TCP analysis draft; avoiding confusion on use Date: Wed, 19 Feb 2003 12:55:41 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Caitlin actually quoted the RFC 1122 text that explains why PUSH won't get the job done: > > The PSH bit is not a record marker and is independent of > > segment boundaries. The transmitter SHOULD collapse > > successive PSH bits when it packetizes data, to send the > > largest possible segment. The problem is that if there is a transmit queue larger than the segment size, existing TCP stacks collapse the multiple PUSH requests into one at the tail of the transmit queue, and then (re)segment the queue into maximally sized segments for transmission, breaking sender alignment. This behavior has been pointed out previously on the tsvwg list. > MPA/TCP overrides that recomendation *for the purpose of > improving performance*. In my mind this is precisely the > type of justification that is required to justify overriding > a SHOULD. This "justify overriding a SHOULD" is the sort of discussion that I seem to recall asking people to avoid. The TSV ADs and the tsvwg WG own this decision, and I would think it's clear by now that their views on this matter may not match those of the MPA advocates. > The criteria has always been interoperability. MPA/TCP does > not do anything to a TCP stream that any ULP has not been > free to do all along. Therefore it does not change TCP. Sorry, incorrect in several ways, and a waste of time ... unless the preferred outcome is to repeat the Atlanta experience in San Francisco and see the rddp group charter cancelled or revised to remove TCP support ... --David > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Wednesday, February 19, 2003 11:05 AM > To: rddp@ietf.org > Subject: RE: [rddp] MPA TCP analysis draft; avoiding confusion on use > > > On 2/19/03, David Borman wrote: > > > >> > >> > >> MPA 5.1.1's SHOULD can be implemented using a 4.2.4.3 > >> compliant flush call. Therefore it does contradict RFC > 1122. > > > >Perhaps you meant "does not contradict"? But no matter, > >because the FLUSH call *throws away* any unsent data (the > >data to the right of the send window). I don't think that > >is what you want. > > > >You can implement sender alignment over an unmodified TCP > >stack. Unfortunately, the most portable way to do that is > >to have application level per-segment acks, and only allow > >one outstanding segment at a time. Thus, it'd probably be > >a performance hit. (Another implementation could be done > >if you could query the TCP stack to see how much data is > >waiting to be ACKed, and just wait for that to go to zero > >before writing the next segment.) > > > > Oops, up too late yesterday. > > The section I should have cited was > > >4.2.2.2 Use of Push: RFC-793 Section 2.8 > > When an application issues a series of SEND calls without > > setting the PUSH flag, the TCP MAY aggregate the data > > internally without sending it. Similarly, when a series > > of segments is received without the PSH bit, a TCP MAY > > queue the data internally without passing it to the > > receiving application. > > > > The PSH bit is not a record marker and is independent of > > segment boundaries. The transmitter SHOULD collapse > > successive PSH bits when it packetizes data, to send the > > largest possible segment. > > > > A TCP MAY implement PUSH flags on SEND calls. If PUSH > > flags are not implemented, then the sending TCP: (1) must > > not buffer data indefinitely, and (2) MUST set the PSH > > bit in the last buffered segment (i.e., when there is no > > more queued data to be sent). > > > > The discussion in RFC-793 on pages 48, 50, and 74 > > erroneously implies that a received PSH flag must be > > passed to the application layer. Passing a received PSH > > flag to the application layer is now OPTIONAL . > > > > An application program is logically required to set the > > PUSH flag in a SEND call whenever it needs to force > > delivery of the data to avoid a communication deadlock. > > However, a TCP SHOULD send a maximum-sized segment > > whenever possible, to improve performance (see Section > > 4.2.3.4 ). > > Note that it does not send TCP MUST send a maximum-sized > segment whenever possible to improve performance. > > MPA/TCP overrides that recomendation *for the purpose of > improving performance*. In my mind this is precisely the > type of justification that is required to justify overriding > a SHOULD. > > An argument could be made that the use of this facility is > not what was intended when the RFC was drafted. But by that > criteria a vast majority of the Web is RFC non-compliant. > > The criteria has always been interoperability. MPA/TCP does > not do anything to a TCP stream that any ULP has not been > free to do all along. Therefore it does not change TCP. > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Feb 19 13:57:19 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04986 for ; Wed, 19 Feb 2003 13:57:19 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1JJ3Uq01668 for rddp-archive@odin.ietf.org; Wed, 19 Feb 2003 14:03:30 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JJ3Up01665 for ; Wed, 19 Feb 2003 14:03:30 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04968 for ; Wed, 19 Feb 2003 13:56:47 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JJ2Hp01632; Wed, 19 Feb 2003 14:02:17 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1JJ1gp01600 for ; Wed, 19 Feb 2003 14:01:42 -0500 Received: from dmz1.silverbacksystems.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA04881 for ; Wed, 19 Feb 2003 13:54:58 -0500 (EST) Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id C8FD7CDCC; Wed, 19 Feb 2003 10:58:33 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id 9DF01172B4; Wed, 19 Feb 2003 10:58:33 -0800 (PST) Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="iso-8859-1" Date: Wed, 19 Feb 2003 10:58:26 -0800 From: "Somesh Gupta" Importance: Normal In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C8E6@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 11102-6320D2E2; Wed, 19 Feb 2003 10:58:15 -0800 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id 2BEB5190E5; Wed, 19 Feb 2003 10:58:15 -0800 (PST) Subject: RE: [rddp] Reality Check: Avoiding confusion on MPA TCP use To: , X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit > -----Original Message----- > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > Black_David@emc.com > Sent: Wednesday, February 19, 2003 9:49 AM > To: rddp@ietf.org > Subject: [rddp] Reality Check: Avoiding confusion on MPA TCP use > Importance: High > > > I need to be more careful about what I wish for - I forgot how > enthusiastic some folks can be about discussing TCP framing and the > IETF process issues that surround it (a momentary lapse - it won't > happen again, at least not anytime soon ;-) ). This message is an > attempt to sort out some of issues raised recently on the list > and/or in private and suggest a productive path forward. > > Let me start with an attempt at a consensus call: > > I believe the rough consensus of the RDDP WG is to pursue > an RDDP over TCP solution that allows cooperating TCP > implementations to employ sender alignment, provided that: > - Alignment is optional > - Receiver support/usage of alignment does not lock out sender > implementations that use unmodified TCP stacks. There was a previous discussion of a protocol design objective here that it not prevent optimization of (i.e. allow implementations that optimize for) a. unmodified TCP senders or b. MPA aware TCP senders > > I hope this is not controversial, but send objections to the list > (as is usual for any attempted consensus call). > > In case anyone finds the above to be at odds with some of what I've > posted recently, all of my comments along the lines of "doing > would require removal of sender alignment from MPA" have been with > the intent of identifying unproductive courses of action (i.e., I > know the MPA advocates don't want to remove alignment from MPA, > hence the intended inference from a statement like "doing would > require removal of alignment" is: "doing would be a bad idea"). > > Assuming the acceptance of the above attempt at consensus, the > following three items are *facts*, and should not be discussed > further on this list: > > (1) Applications that use TCP don't make changes to TCP. Sender > alignment is a TCP change, so MPA is not just an application. > (2) The "proof" requirement is linked to the TCP sender alignment > change, so the above rough consensus entails providing a "proof". > (3) The Transport ADs hold all the cards, specifically the RDDP WG > charter is at high risk of being withdrawn or modified to exclude > TCP if the WG insists on extensive exploration of the tarpits > of whether TCP is being changed and whether "proof" is necessary. > > Those with questions/concerns/objections to the above are welcome to > contact me directly off the list, but the above three items are not > open to change and should not be discussed further on the list > (yes, the ADs are watching ...). > > Turning to a more positive topic, there is a path forward available > in the Elzur MPA TCP analysis draft. Section 8.1 says: > > The third model is by negotiation of MPA transition by the ULP > using octet stream messages to accomplish a ULP specific > , , three way exchange. > > If this exchange were made MPA-generic instead of ULP-specific and > applied to all transitions to MPA, including those that occur > immediately at the start of a connection, then Figure 5 in that > draft contains much of the "proof" structure and content. > > This approach would also provide a means of dealing with the > current unrealistic buffer size analysis in Section 7. The > problem is that *if*, as a number of people have posted, a > receiver cannot tell the difference between the bytestream > arriving from an unmodified TCP vs. an MPA-enabled TCP, then > a receiver has to assume the former, and cannot achieve the > desired "BufferSizeAF = K2 * EMSS". FWIW, the approach of > requiring that unmodified TCPs and MPA-enabled TCPs contact > receivers on different ports is unlikely to be acceptable, and > kicking transport buffer sizing issues up to ULPs to deal with > is silly. > > In addition, this sort of exchange would also provide a means of > making the current 512-byte fixed marker interval variable, which > is probably going to be needed to seriously deal with unmodified > TCP stacks (e.g., are 16 syscalls or other send events/descriptors > really reasonable to send an 8k DDP data block?). > > If I can obtain rough consensus to proceed in this direction > (initial MPA-generic Hello/Hello ACK/ACK exchange used in all cases), > I can then consult with the ADs on the strength of "proof" that the > design of the MPA Hello/Hello ACK/ACK needs to provide. > > At the moment, I don't understand the level of emotion surrounding > the opposition to some sort of initial negotiation. I was under > the impression that RDDP connections were likely to be long-lived > "elephants" rather than short-lived "mice" and that the networks > of primary interest were low-latency (e.g., data center). If > that's the case, the latency of the extra round trip and overhead > of the extra bytes transferred are going to vanish into the noise > (of the three messages, only MPA Hello can't carry data) - what > have I missed? > > As indicated above, any discussion of items (1) - (3) should be > sent to me directly off the list. Discussion of MPA-generic > negotiation is welcome, but objections to it should suggest > alternate means of providing the "proof" and providing > receivers with information about sender alignment behavior. > > I really am trying to move this forward, and I believe sender > alignment can be part of the solution, but we need to play the > hand we've been dealt, rather than the one we'd like to have. > > Thanks, > --David > > ---------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > black_david@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Feb 19 19:12:42 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA13663 for ; Wed, 19 Feb 2003 19:12:42 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1K0J0w21363 for rddp-archive@odin.ietf.org; Wed, 19 Feb 2003 19:19:00 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1K0J0p21360 for ; Wed, 19 Feb 2003 19:19:00 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA13660 for ; Wed, 19 Feb 2003 19:12:11 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1K0IJp21350; Wed, 19 Feb 2003 19:18:19 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1K0HKp21330 for ; Wed, 19 Feb 2003 19:17:20 -0500 Received: from ztxmail04.ztx.compaq.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA13635 for ; Wed, 19 Feb 2003 19:10:31 -0500 (EST) Received: from cceexg12.americas.cpqcorp.net (cceexg12.americas.cpqcorp.net [16.110.250.124]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id C0384496D; Wed, 19 Feb 2003 18:14:19 -0600 (CST) Received: from cceexc16.americas.cpqcorp.net ([16.110.250.58]) by cceexg12.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Wed, 19 Feb 2003 18:14:19 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: RE: [rddp] Reality Check: Avoiding confusion on MPA TCP use Date: Wed, 19 Feb 2003 18:14:19 -0600 Message-ID: <01CA1B1F74A4EC40A9C887F7B58F8724031430DA@cceexc16.americas.cpqcorp.net> Thread-Topic: [rddp] Reality Check: Avoiding confusion on MPA TCP use Thread-Index: AcLYQRs2i44EpOGbTsKubnjysu0DzwALb8KQ From: "Barron, Dwight" To: , X-OriginalArrivalTime: 20 Feb 2003 00:14:19.0626 (UTC) FILETIME=[033344A0:01C2D875] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h1K0HKp21331 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit David, I am only copying specific excerpts from your message in hopes that ensuing discussions on this thread stay focused on this thread topic, "Avoiding confusion on MPA TCP use." I believe the objections to having an MPA-generic exchange are a) where/how to unambiguously insert it into a TCP bytestream and b) how does that significantly improve the error quadrants in Fig 5 of Draft-Elzur? This is based on the assumption that a ULP that wants to use DDP/MPA will explicitly know of the opportunity and treat the transition as a command operation rather than a data transfer operation. Only the ULP can safely deduce when and how it is appropriate to insert such a command operation in the TCP bytestream. Note that the position of this command in the bytestream is dictated by the ULP's own protocol design (e.g. iSCSI, SMB, DNS) and is independent of any form of transmitter alignment, PUSH flags and other whatnot. Requiring the sequence as the first bytes transmitted on a connection does not significantly improve the error detection characteristics in the upper right quadrant of Fig 5. Given your example of the difficulty knowing all ! the private protocols running on TCP, I don't believe we can design a provably correct sequence that would never confuse any existing or future TCP application. What we can provably do is show that very little forward progress will be made before the an error is detected, due primarily to the lower left quadrant of Fig 5. I also believe that we can show that a ULP specific transition-to-MPA-command, when injected into the ULP-specific command portion of the TCP bytestream, is at no greater risk of being mis-interpreted than any other ULP-specific command and that this is an equal or lower risk of error than an MPA-generic hello sequence that is equally mis-configured. However, if this is an open book test and you have already given us the "correct" answer, we can craft a generic and turn in our test books. Regards, Dwight > Turning to a more positive topic, there is a path forward available > in the Elzur MPA TCP analysis draft. Section 8.1 says: > > The third model is by negotiation of MPA transition by the ULP > using octet stream messages to accomplish a ULP specific > , , three way exchange. > > If this exchange were made MPA-generic instead of ULP-specific and > applied to all transitions to MPA, including those that occur > immediately at the start of a connection, then Figure 5 in that > draft contains much of the "proof" structure and content. > > .... snip > > If I can obtain rough consensus to proceed in this direction > (initial MPA-generic Hello/Hello ACK/ACK exchange used in all cases), > I can then consult with the ADs on the strength of "proof" that the > design of the MPA Hello/Hello ACK/ACK needs to provide. > > At the moment, I don't understand the level of emotion surrounding > the opposition to some sort of initial negotiation. I was under > the impression that RDDP connections were likely to be long-lived > "elephants" rather than short-lived "mice" and that the networks > of primary interest were low-latency (e.g., data center). If > that's the case, the latency of the extra round trip and overhead > of the extra bytes transferred are going to vanish into the noise > (of the three messages, only MPA Hello can't carry data) - what > have I missed? _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Feb 19 21:03:31 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA15564 for ; Wed, 19 Feb 2003 21:03:31 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1K29pI27427 for rddp-archive@odin.ietf.org; Wed, 19 Feb 2003 21:09:51 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1K29pp27424 for ; Wed, 19 Feb 2003 21:09:51 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA15543 for ; Wed, 19 Feb 2003 21:02:59 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1K29Ap27406; Wed, 19 Feb 2003 21:09:10 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1K28xp27375 for ; Wed, 19 Feb 2003 21:08:59 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA15530 for ; Wed, 19 Feb 2003 21:02:07 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRKNXNM>; Wed, 19 Feb 2003 21:05:57 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C8F2@corpmx14.us.dg.com> To: Dwight.Barron@hp.com, rddp@ietf.org Subject: RE: [rddp] Reality Check: Avoiding confusion on MPA TCP use Date: Wed, 19 Feb 2003 21:03:39 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Dwight, > I believe the objections to having an MPA-generic > exchange are a) where/how to unambiguously insert it into a > TCP bytestream and b) how does that significantly improve the > error quadrants in Fig 5 of Draft-Elzur? My primary concern is with improving the upper right hand error quadrant in Figure 4 of Draft-Elzur. I suggested an MPA-generic mechanism based on Figure 5 in the hope of using the same mechanism in both cases, but that's not necessary. > This is based on the > assumption that a ULP that wants to use DDP/MPA will > explicitly know of the opportunity and treat the transition > as a command operation rather than a data transfer operation. > Only the ULP can safely deduce when and how it is appropriate > to insert such a command operation in the TCP bytestream. > Note that the position of this command in the bytestream is > dictated by the ULP's own protocol design (e.g. iSCSI, SMB, > DNS) and is independent of any form of transmitter alignment, > PUSH flags and other whatnot. The ULP could send the command in non-MPA mode and then wait until some sort of ULP response is received from the other side before instructing the RNIC to go into MPA mode (send , etc.) on that connection. OTOH, my primary concern is with the upper right quadrant in Figure 4, so for the Figure 5 case, I agree that: > Requiring the > sequence as the first bytes transmitted on a connection does > not significantly improve the error detection characteristics > in the upper right quadrant of Fig 5. except that the won't necessarily be the first bytes transmitted on the connection in the Figure 5 case. OTOH, if Figure 4 was intended in the above, I disagree. So, while using the same mechanism independent of when a connection goes into MPA mode has the virtue of consistency and might simplify things, doing so is in the domain of "nice to have", as opposed to an absolute requirement. An alternative design approach is fine provided that the rationale for the design selected is solid and well-documented, and, in addition, there is adequate implementation guidance to make it likely that a ULP implementer who is not part of this community will get the synchronization aspects of the transition to MPA right (these provisos apply no matter what design approach is selected). For example, sending a ULP "" command and immediately following it with an MPA-based transmission may cause problems if the receiver has to retrieve the command before it can switch into MPA mode on that connection, as by then, the first MPA-based transmission may have passed through without any MPA-based processing. Section 8.1 of the MPA draft falls short in the area of guidance to implementers. In addition, it will be necessary to address the means of ascertaining sender behavior so that the BufferAF level of required buffering resources (Section 7 of Draft-Elzur) can be achieved. It appears that Draft-Elzur envisions passing of explicit ULP and the like tokens, rather than allowing a ULP to determine the MPA start point implicitly via other means. If that was intended as a requirement for all connections that transition into MPA mode other than at initial establishment, it'll need to go into the main MPA draft as a "MUST" requirement imposed on ULP designers. When the full description of the ULP-based mechanism including both full ULP requirements (e.g., cf. immediately preceding paragraph) and ULP implementation guidance/recommendations are available, it will be possible to ascertain the level of assurance it provides on getting the transition to MPA correct in the Figure 5 case. > Given your example of > the difficulty knowing all the private protocols running on > TCP, I don't believe we can design a provably correct Hello> sequence that would never confuse any existing or > future TCP application. It seems that "provably correct" is being routinely over-interpreted as requiring "absolute proof of correctness in all circumstances". As I've said, not even security protocols can reach that level of assurance, and I would hope that the level of assurance required here will be less than that demanded of security protocols. OTOH, Section 8.2 of Draft-Elzur ("we spent some time looking for a counterexample and didn't find one") is insufficient, and the size of Section 8.2 by comparison to Figure 5 is indicative of the relative ability of the two approaches to provide a concise "proof". > However, if this is an open book test and you have already > given us the "correct" answer, we can craft a generic Hello> and turn in our test books. Ah, but this is one of those truly annoying test questions that doesn't have a single right answer :-), and I make no claim to have provided the "best" answer. OTOH, I believe this approach to be workable. Under ordinary circumstances, I wouldn't do this level of design as a WG Chair, but I've received complaints that the design task being set for the WG in this area is impossible, and hence I've described a design approach as a counterexample to demonstrate that the task is feasible. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Feb 19 21:11:19 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA15658 for ; Wed, 19 Feb 2003 21:11:19 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1K2HeV27645 for rddp-archive@odin.ietf.org; Wed, 19 Feb 2003 21:17:40 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1K2Hep27642 for ; Wed, 19 Feb 2003 21:17:40 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA15648 for ; Wed, 19 Feb 2003 21:10:47 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1K2H1p27619; Wed, 19 Feb 2003 21:17:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1K2Gcp27598 for ; Wed, 19 Feb 2003 21:16:38 -0500 Received: from mxic2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA15622 for ; Wed, 19 Feb 2003 21:09:46 -0500 (EST) From: Black_David@emc.com Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRHP718>; Wed, 19 Feb 2003 21:13:31 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C8F3@corpmx14.us.dg.com> To: rddp@ietf.org Date: Wed, 19 Feb 2003 21:11:13 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: multipart/mixed; boundary="----_=_NextPart_000_01C2D885.5804BA30" Subject: [rddp] FW: I-D ACTION:draft-williams-iwarp-ift-01.txt Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This message is in MIME format. Since your mail reader does not understand this format, some or all of this message may not be legible. ------_=_NextPart_000_01C2D885.5804BA30 Content-Type: text/plain; charset="iso-8859-1" FYI, although this contains an initial adaption exchange mechanism, I did not put Jim up to including one. Thanks, --David -----Original Message----- From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org] Sent: Wednesday, February 19, 2003 10:05 AM Subject: I-D ACTION:draft-williams-iwarp-ift-01.txt A New Internet-Draft is available from the on-line Internet-Drafts directories. Title : iWARP Framing for TCP Author(s) : J. Williams Filename : draft-williams-iwarp-ift-01.txt Pages : 10 Date : 2003-2-18 A framing protocol is defined for DDP over TCP that is fully compliant with applicable TCP RFCs and fully interoperable with existing TCP implementations. This protocol is being offered as an alternative to to the proposal offered in [MPA]. The protocol offers to two things, definition of the DDP record boundaries within the TCP stream, and added data integrity by means of added CRC protection. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-williams-iwarp-ift-01.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-williams-iwarp-ift-01.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-williams-iwarp-ift-01.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. ------_=_NextPart_000_01C2D885.5804BA30 Content-Type: message/rfc822 To: Subject: Date: Wed, 19 Feb 2003 21:13:31 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: multipart/mixed; boundary="----_=_NextPart_002_01C2D885.5804BA30" ------_=_NextPart_002_01C2D885.5804BA30 Content-Type: text/plain ------_=_NextPart_002_01C2D885.5804BA30 Content-Type: application/octet-stream; name="ATT61053" Content-Disposition: attachment; filename="ATT61053" Content-type: message/external-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <2003-2-18160346.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-williams-iwarp-ift-01.txt ------_=_NextPart_002_01C2D885.5804BA30 Content-Type: message/external-body; site="internet-drafts"; dir="draft-williams-iwarp-ift-01.txt"; mode="ftp.ietf.org"; access-type="anon-ftp" ------_=_NextPart_002_01C2D885.5804BA30-- ------_=_NextPart_000_01C2D885.5804BA30-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 20 07:48:40 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA10255 for ; Thu, 20 Feb 2003 07:48:40 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1KCtD909628 for rddp-archive@odin.ietf.org; Thu, 20 Feb 2003 07:55:13 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1KCtDp09625 for ; Thu, 20 Feb 2003 07:55:13 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA10087 for ; Thu, 20 Feb 2003 07:48:08 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1KCsEp09509; Thu, 20 Feb 2003 07:54:14 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1KCjap09014 for ; Thu, 20 Feb 2003 07:45:36 -0500 Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA09554; Thu, 20 Feb 2003 07:38:33 -0500 (EST) Message-Id: <200302201238.HAA09554@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: rddp@ietf.org From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Date: Thu, 20 Feb 2003 07:38:33 -0500 Subject: [rddp] I-D ACTION:draft-ietf-rddp-rdmap-00.txt Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Remote Direct Data Placement Working Group of the IETF. Title : An RDMA Protocol Specification Author(s) : R. Recio et al. Filename : draft-ietf-rddp-rdmap-00.txt Pages : 66 Date : 2003-2-19 This document defines a Remote Direct Memory Access Protocol (RDMAP) that operates over the Direct Data Placement Protocol (DDP protocol). RDMAP provides read and write services directly to applications and enables data to be transferred directly into ULP Buffers without intermediate data copies. It also enables a kernel bypass implementation. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-rddp-rdmap-00.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-rddp-rdmap-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-rddp-rdmap-00.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <2003-2-19145949.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-rddp-rdmap-00.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-rddp-rdmap-00.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <2003-2-19145949.I-D@ietf.org> --OtherAccess-- --NextPart-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 20 11:51:58 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA21068 for ; Thu, 20 Feb 2003 11:51:58 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1KGwcO28409 for rddp-archive@odin.ietf.org; Thu, 20 Feb 2003 11:58:38 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1KGwbp28406 for ; Thu, 20 Feb 2003 11:58:37 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA21050 for ; Thu, 20 Feb 2003 11:51:27 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1KGvap28303; Thu, 20 Feb 2003 11:57:36 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1KGu2p28209 for ; Thu, 20 Feb 2003 11:56:02 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA20938 for ; Thu, 20 Feb 2003 11:48:52 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 6747F246968 for ; Thu, 20 Feb 2003 11:52:20 -0500 (EST) Date: Thu, 20 Feb 2003 10:53:35 -0600 From: Caitlin Bestler To: rddp@ietf.org X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] re: draft-elzur - Partial CRC Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit draft-elzue in section 4 makes the assumption that CRC32c must be calculatd serially. Section 4.a.ii.1 states: > When a complete FPDU is available, a similar procedure to > the in-order algorithm above is used. There is additional > complexity, though, because when the missing segment > arrives, this TCP segment must be run through the CRC > engine after the CRC is calculated for the missing > segment. While this is a vaid implementation, it is not required. Each bit in the CRC32c for the FPDU is an XOR of a set of the covered bits within the FPDU. XOR is a commutative operation, hence the FPDU's CRC32c does not need to be calculated in order. However, the buffering benefits cited for aligned FPDUs remain valid. Additionally, a serial CRC32c calculation is simpler to implemenet than a random one. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 20 13:39:40 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA24815 for ; Thu, 20 Feb 2003 13:39:40 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1KIkKT05188 for rddp-archive@odin.ietf.org; Thu, 20 Feb 2003 13:46:20 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1KIkKp05185 for ; Thu, 20 Feb 2003 13:46:20 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA24784 for ; Thu, 20 Feb 2003 13:39:08 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1KIiKp05043; Thu, 20 Feb 2003 13:44:20 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1KIhAp04969 for ; Thu, 20 Feb 2003 13:43:10 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA24613 for ; Thu, 20 Feb 2003 13:35:58 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 131492492BF for ; Thu, 20 Feb 2003 13:39:50 -0500 (EST) Date: Thu, 20 Feb 2003 12:41:03 -0600 From: Caitlin Bestler To: rddp@ietf.org X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] Problems with MPA Hello Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit The introduction of an MPA Hello offers no actual solutions to any real problems. It merely complicates the protocols, and lengthens the minimum time for connection establishment. Point 1: MPA Mode can only be initiated by the ULP. An MPA Hello cannot be used to initialize the MPA state. The most it can do is confirm the the peer ULP's intent. Rationale: Under existing RFCs, an incoming TCP stream may consist of any sequence of bytes. There are no restrictions. Period. If MPA cannot "alter" TCP by specifying segmentation points that would have been legal anyway, then it *certainly* cannot place restrictions on what can be a TCP stream's payload. A combined receicer that support Non-MPA and MPA over TCP MUST NOT enter MPA mode for any TCP connection prior to the point in the stream authorized by the local ULP. Point 2: There is no such thing as an aligned TCP stream. There are only TCP streams that are currently alligned. The only information the the sending MPA layer has that the sending ULP lacks is whether the sending MPA will align the TCP segments it generates with FPDUs. It CANNOT know that there will be no middlebox that will resegment the TCP stream for the simple reason that it CANNOT know what set of middleboxes will deliver the IP datagram. Just as a reciever MUST be prepared to handle PMTU changes, it MUST be prepared to handle "unexpected" drops in alignment. More fundamentally, I would submit that for MPA to claim forward knowledge of the behavior of an IP network is a violation of the most basic IP principles. If such network knowledge were available and legitimate to use, RDDP would benefit far more by simply ensuring that ALL DDP Segments were sized to conform with the *future* PMTU. Point 3: Whether or not the sender MPA aligns is at best a hint to the receiver. Niether the set of packets that will be accepted, nor the ULP required response to them, is affected by alignment. A server MUST NOT drop a connection because it unexpectedly changes PMTU or drops alignment. Point 4: There is no information for the sender MPA to add to an MPA Hello. The sending MPA layer has no data that can be placed in an MPA Hello that was not already available to both ULPs. Point 5: The ULPs are already responsible for ULP matching. As with every other IP protocol ever created, it is a ULP responsibility to know that its peer is complying wth the same protocol, and to take reasonable protections from both accidental connections and malicious attacks. The use of MPA/TCP is already implicit in the identity of the ULP itself. There are no ULP protocols that work over *either* TCP or RDDP. Point 6: A confirming MPA Hello delays processing of other DDP segments received out-of-order. If the purpose of an MPA Hello is to confirm the operative MPA mode *before* any MPA processing then all segments received out of order must be held for deferred processing. Point 7: The MPA Hello message wastes an entire TCP segment or violates DDP / LLP layering. An MPA Hello message reduces the space available in the first LLP message. The DDP layer will be relying upon the normal PMTU when it creates DDP Segments. These will not fit in the same segment IF DDP layer reduces the size of the first segment to accomodate the MPA Hello then it is violating layering, by modifying its behavior based upon knowledge of the LLP. Unless we want to modify the LLP/DDP interface to report a suggested DDP segment size on a per request basis there is no way to accomodate this exception in an LLP neutral fashion. For a *new* connection, especially a long haul one, wasting a single TCP segment is signifigant. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 20 17:30:48 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA02141 for ; Thu, 20 Feb 2003 17:30:48 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1KMbYU22637 for rddp-archive@odin.ietf.org; Thu, 20 Feb 2003 17:37:34 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1KMbYp22634 for ; Thu, 20 Feb 2003 17:37:34 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA02120 for ; Thu, 20 Feb 2003 17:30:16 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1KMa8p21898; Thu, 20 Feb 2003 17:36:08 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1KMZxp21884 for ; Thu, 20 Feb 2003 17:35:59 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA02092 for ; Thu, 20 Feb 2003 17:28:42 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRKQSDT>; Thu, 20 Feb 2003 17:32:33 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C8FC@corpmx14.us.dg.com> To: cait@asomi.com, rddp@ietf.org Subject: RE: [rddp] Problems with MPA Hello Date: Thu, 20 Feb 2003 17:30:13 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Caitlin, > Point 2: There is no such thing as an aligned TCP stream. > There are only TCP streams that are currently alligned. > > Point 3: Whether or not the sender MPA aligns is at best > a hint to the receiver. Unfortunately, neither of these Points is consistent with the buffering reduction discussion in Section 7 of Draft-Elzur which is about a receiver allocating worst case resources on the assumption that an aligned stream will stay aligned. If your above two Points are correct, that discussion of buffer optimization for aligned senders ought to be called out in the MPA draft proper as something that an MPA receiver SHOULD NOT do for robustness reasons. Somehow, I don't think that's what the MPA authors had in mind ... but, if you can convince them to add such a "SHOULD NOT", that would be progress. I'll wait to see how that turns out before venturing further into the other Points. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Thursday, February 20, 2003 1:41 PM > To: rddp@ietf.org > Subject: [rddp] Problems with MPA Hello > > > The introduction of an MPA Hello offers no actual solutions > to any real problems. It merely complicates the protocols, > and lengthens the minimum time for connection establishment. > > > Point 1: MPA Mode can only be initiated by the ULP. > > An MPA Hello cannot be used to initialize the MPA state. > The most it can do is confirm the the peer ULP's intent. > > Rationale: Under existing RFCs, an incoming TCP stream > may consist of any sequence of bytes. There are no > restrictions. Period. If MPA cannot "alter" TCP by > specifying segmentation points that would have been legal > anyway, then it *certainly* cannot place restrictions on > what can be a TCP stream's payload. > > A combined receicer that support Non-MPA and MPA over TCP > MUST NOT enter MPA mode for any TCP connection prior to > the point in the stream authorized by the local ULP. > > > > > Point 2: There is no such thing as an aligned TCP stream. > There are only TCP streams that are currently alligned. > > The only information the the sending MPA layer has that > the sending ULP lacks is whether the sending MPA will > align the TCP segments it generates with FPDUs. > > It CANNOT know that there will be no middlebox that will > resegment the TCP stream for the simple reason that it > CANNOT know what set of middleboxes will deliver the > IP datagram. > > Just as a reciever MUST be prepared to handle PMTU changes, > it MUST be prepared to handle "unexpected" drops in > alignment. > > More fundamentally, I would submit that for MPA to claim > forward knowledge of the behavior of an IP network is a > violation of the most basic IP principles. If such network > knowledge were available and legitimate to use, RDDP would > benefit far more by simply ensuring that ALL DDP Segments > were sized to conform with the *future* PMTU. > > > > Point 3: Whether or not the sender MPA aligns is at best > a hint to the receiver. > > Niether the set of packets that will be accepted, nor the > ULP required response to them, is affected by alignment. > > A server MUST NOT drop a connection because it unexpectedly > changes PMTU or drops alignment. > > > > Point 4: There is no information for the sender MPA to add > to an MPA Hello. > > The sending MPA layer has no data that can be placed in an > MPA Hello that was not already available to both ULPs. > > > > > Point 5: The ULPs are already responsible for ULP matching. > > As with every other IP protocol ever created, it is a ULP > responsibility to know that its peer is complying wth the > same protocol, and to take reasonable protections from both > accidental connections and malicious attacks. > > The use of MPA/TCP is already implicit in the identity of > the ULP itself. There are no ULP protocols that work over > *either* TCP or RDDP. > > > > > Point 6: A confirming MPA Hello delays processing of > other DDP segments received out-of-order. > > If the purpose of an MPA Hello is to confirm the operative > MPA mode *before* any MPA processing then all segments > received out of order must be held for deferred processing. > > > > > Point 7: The MPA Hello message wastes an entire TCP segment > or violates DDP / LLP layering. > > An MPA Hello message reduces the space available in the > first LLP message. The DDP layer will be relying upon the > normal PMTU when it creates DDP Segments. These will not > fit in the same segment > > IF DDP layer reduces the size of the first segment to > accomodate the MPA Hello then it is violating layering, by > modifying its behavior based upon knowledge of the LLP. > Unless we want to modify the LLP/DDP interface to report > a suggested DDP segment size on a per request basis there > is no way to accomodate this exception in an LLP neutral > fashion. > > For a *new* connection, especially a long haul one, wasting > a single TCP segment is signifigant. > > > > > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 20 18:56:45 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA03965 for ; Thu, 20 Feb 2003 18:56:45 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1L03YT27301 for rddp-archive@odin.ietf.org; Thu, 20 Feb 2003 19:03:34 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L03Xp27298 for ; Thu, 20 Feb 2003 19:03:33 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA03941 for ; Thu, 20 Feb 2003 18:56:14 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1KNx8p27071; Thu, 20 Feb 2003 18:59:08 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1KNwCp27021 for ; Thu, 20 Feb 2003 18:58:12 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA03782 for ; Thu, 20 Feb 2003 18:50:53 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 313482473F4; Thu, 20 Feb 2003 18:54:46 -0500 (EST) Date: Thu, 20 Feb 2003 17:56:00 -0600 From: Caitlin Bestler Subject: RE: [rddp] Problems with MPA Hello To: Black_David@emc.com, rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C8FC@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/20/03, Black_David@emc.com wrote: >Caitlin, > >> Point 2: There is no such thing as an aligned TCP stream. >> There are only TCP streams that are currently alligned. >> >> Point 3: Whether or not the sender MPA aligns is at best >> a hint to the receiver. > >Unfortunately, neither of these Points is consistent with >the buffering reduction discussion in Section 7 of >Draft-Elzur which is about a receiver allocating worst >case resources on the assumption that an aligned stream >will stay aligned. > >If your above two Points are correct, that discussion of >buffer optimization for aligned senders ought to be called >out in the MPA draft proper as something that an MPA >receiver SHOULD NOT do for robustness reasons. Somehow, I >don't think that's what the MPA authors had in mind ... >but, if you can convince them to add such a "SHOULD NOT", >that would be progress. I'll wait to see how that turns >out before venturing further into the other Points. > This topic has already been thoroughly covered on the reflector, but I'll go ahead and recap: - Resegmentation is legal for TCP-aware middleboxes. - The set of middleboxes that an IP datagram will traverse should not be known by the ULP in advance. As that the ULP has no means of physically preventing a network re-configuration from occurring, it also cannot do so. - Therefore it is possible to lose alignment "suddenly". The ULP could be halfway through committing some file or database changes when this happens. - However it will be rare. - An RNIC could make a statistical estimate of the resources required to handle buffering caused by loss of alignment. If those resources are exceeded, it will drop the IP datagrams that required the exhausted resources. This is no different than what any of the switches, routers and other middleboxes between the two RNICs could do if *their* statistical estimates are exceeded. Obviously, just as with the middleboxes, the statistical estimate SHOULD be heavily on the safe side. How safe is a market issue, not an IETF issue. - MPA/TCP is still a reliable transport protocol. It will recover from IP datagrams being dropped due to resource exhaustion. The validity of the ULP connection will not be broken. - An RNIC also has the option of "slow pathing" connections via the host memory itself. Of course this is ultimately just another form of buffering, and it can also be exhausted so the point has not really changed. But it does emphasize that requiring an RNIC to be robust in the face of lost alignment DOES NOT require the addition of vast memory resources on the RNIC itself. Loss of alignment *will* have a throughput impact. So do Ethernet collisions and Router-dropped packets. The point of a reliable transport layer is to be robust in the face of such errors. If you value *guaranteed* speed over reliability, then you should be using an unreliable transport. If we want to design something that only works in a Data Center then why don't we call it IBoE (InfiniBand over Ethernet). Since the goal is RDMA over IP, then we had best accept the behavior of IP networks, complete with the need to deal with statistical behavior, and design accordingly. Requiring an MPA/TCP receiver to remain robust in the event of alignment loss is what is required to truly be a reliable IP transport protocol. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 20 19:39:58 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA04870 for ; Thu, 20 Feb 2003 19:39:58 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1L0kjj30437 for rddp-archive@odin.ietf.org; Thu, 20 Feb 2003 19:46:45 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L0kjp30434 for ; Thu, 20 Feb 2003 19:46:45 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA04859 for ; Thu, 20 Feb 2003 19:39:26 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L0k3p30404; Thu, 20 Feb 2003 19:46:03 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L0jtp30384 for ; Thu, 20 Feb 2003 19:45:55 -0500 Received: from dmz1.silverbacksystems.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA04852 for ; Thu, 20 Feb 2003 19:38:36 -0500 (EST) Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id 902E9D164; Thu, 20 Feb 2003 16:42:13 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id 5BC8B194E1; Thu, 20 Feb 2003 16:42:13 -0800 (PST) Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii" Date: Thu, 20 Feb 2003 16:42:09 -0800 From: "Somesh Gupta" Importance: Normal In-Reply-To: Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 03602-37610FE2; Thu, 20 Feb 2003 16:41:57 -0800 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id AFC58194CA; Thu, 20 Feb 2003 16:41:57 -0800 (PST) Subject: RE: [rddp] Problems with MPA Hello To: "Caitlin Bestler" , , X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Caitlin, Comment in line and then at the end. > -----Original Message----- > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > Caitlin Bestler > Sent: Thursday, February 20, 2003 3:56 PM > To: Black_David@emc.com; rddp@ietf.org > Subject: RE: [rddp] Problems with MPA Hello > > > On 2/20/03, Black_David@emc.com wrote: > > >Caitlin, > > > >> Point 2: There is no such thing as an aligned TCP stream. > >> There are only TCP streams that are currently alligned. > >> > >> Point 3: Whether or not the sender MPA aligns is at best > >> a hint to the receiver. > > > >Unfortunately, neither of these Points is consistent with > >the buffering reduction discussion in Section 7 of > >Draft-Elzur which is about a receiver allocating worst > >case resources on the assumption that an aligned stream > >will stay aligned. > > > >If your above two Points are correct, that discussion of > >buffer optimization for aligned senders ought to be called > >out in the MPA draft proper as something that an MPA > >receiver SHOULD NOT do for robustness reasons. Somehow, I > >don't think that's what the MPA authors had in mind ... > >but, if you can convince them to add such a "SHOULD NOT", > >that would be progress. I'll wait to see how that turns > >out before venturing further into the other Points. > > > This topic has already been thoroughly covered on the > reflector, but I'll go ahead and recap: > > - Resegmentation is legal for TCP-aware middleboxes. > > - The set of middleboxes that an IP datagram will traverse > should not be known by the ULP in advance. As that the ULP > has no means of physically preventing a network > re-configuration from occurring, it also cannot do so. > > - Therefore it is possible to lose alignment "suddenly". > The ULP could be halfway through committing some file > or database changes when this happens. > > - However it will be rare. As David previously pointed out (I think), this is not so rare for software based senders. A look at the tcp_output function of bsd tcp (the code after send: label) will help clarify. Any time, data gets queued on the socket head because the sender is sending faster than tcp can send (congestion window is small due to slow start or congestion avoidance or the sender is just plain fast), the alignment will get broken. Of course this may not be easily perceived from traces where a single request/single response at a time model is being followed. However, with RDMA/DDP, there may be many messages outstanding at a time that are not a multiple of segment size and misalignment will happen unless the sender TCP is modified. > > - An RNIC could make a statistical estimate of the > resources required to handle buffering caused by loss of > alignment. If those resources are exceeded, it will drop > the IP datagrams that required the exhausted resources. > This is no different than what any of the switches, routers > and other middleboxes between the two RNICs could do if > *their* statistical estimates are exceeded. Obviously, > just as with the middleboxes, the statistical estimate > SHOULD be heavily on the safe side. How safe is a market > issue, not an IETF issue. > > - MPA/TCP is still a reliable transport protocol. It will > recover from IP datagrams being dropped due to resource > exhaustion. The validity of the ULP connection will not > be broken. > > - An RNIC also has the option of "slow pathing" connections > via the host memory itself. Of course this is ultimately > just another form of buffering, and it can also be exhausted > so the point has not really changed. But it does emphasize > that requiring an RNIC to be robust in the face of lost > alignment DOES NOT require the addition of vast memory > resources on the RNIC itself. > > > Loss of alignment *will* have a throughput impact. I think the issue is whether we will let implementations decide this or the protocol decide this. I feel we should let implementation decide this and design a framing protocol that does not impose this. So do > Ethernet collisions and Router-dropped packets. The point of > a reliable transport layer is to be robust in the face of > such errors. If you value *guaranteed* speed over > reliability, then you should be using an unreliable > transport. > > If we want to design something that only works in a Data > Center then why don't we call it IBoE (InfiniBand over > Ethernet). Since the goal is RDMA over IP, then we had best > accept the behavior of IP networks, complete with the need > to deal with statistical behavior, and design accordingly. Agree. > > Requiring an MPA/TCP receiver to remain robust in the event > of alignment loss is what is required to truly be a reliable > IP transport protocol. If you look at Jim William's proposal, it provides for a TCP compliant framing proposal, where the implementor can decide whether to optimize for aligned senders (like in a data center), legacy senders (where migration is easier such as for iSCSI currently) or blend in both. I think that unless the protocol is targeted at the data center only, such an approach, which should be acceptable as a standards track document, is better. Somesh > > > > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 20 21:58:04 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA07238 for ; Thu, 20 Feb 2003 21:58:04 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1L34td05319 for rddp-archive@odin.ietf.org; Thu, 20 Feb 2003 22:04:55 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L34tp05316 for ; Thu, 20 Feb 2003 22:04:55 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA07234 for ; Thu, 20 Feb 2003 21:57:32 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L34Dp05298; Thu, 20 Feb 2003 22:04:13 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L33Bp05255 for ; Thu, 20 Feb 2003 22:03:11 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA07219 for ; Thu, 20 Feb 2003 21:55:49 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 3FB98245F7F; Thu, 20 Feb 2003 21:59:41 -0500 (EST) Date: Thu, 20 Feb 2003 21:00:56 -0600 From: Caitlin Bestler Subject: RE: [rddp] Problems with MPA Hello To: Somesh Gupta , rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/20/03, Somesh Gupta wrote: >Caitlin, > >Comment in line and then at the end. > >> -----Original Message----- >> From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of >> Caitlin Bestler >> Sent: Thursday, February 20, 2003 3:56 PM >> To: Black_David@emc.com; rddp@ietf.org >> Subject: RE: [rddp] Problems with MPA Hello >> >> >> On 2/20/03, Black_David@emc.com wrote: >> >> >Caitlin, >> > >> >> Point 2: There is no such thing as an aligned TCP stream. >> >> There are only TCP streams that are currently alligned. >> >> >> >> Point 3: Whether or not the sender MPA aligns is at best >> >> a hint to the receiver. >> > >> >Unfortunately, neither of these Points is consistent with >> >the buffering reduction discussion in Section 7 of >> >Draft-Elzur which is about a receiver allocating worst >> >case resources on the assumption that an aligned stream >> >will stay aligned. >> > >> >If your above two Points are correct, that discussion of >> >buffer optimization for aligned senders ought to be called >> >out in the MPA draft proper as something that an MPA >> >receiver SHOULD NOT do for robustness reasons. Somehow, I >> >don't think that's what the MPA authors had in mind ... >> >but, if you can convince them to add such a "SHOULD NOT", >> >that would be progress. I'll wait to see how that turns >> >out before venturing further into the other Points. >> > >> This topic has already been thoroughly covered on the >> reflector, but I'll go ahead and recap: >> >> - Resegmentation is legal for TCP-aware middleboxes. >> >> - The set of middleboxes that an IP datagram will traverse >> should not be known by the ULP in advance. As that the ULP >> has no means of physically preventing a network >> re-configuration from occurring, it also cannot do so. >> >> - Therefore it is possible to lose alignment "suddenly". >> The ULP could be halfway through committing some file >> or database changes when this happens. >> >> - However it will be rare. > > As David previously pointed out (I think), this is not so > rare for software based senders. A look at the tcp_output > function of bsd tcp (the code after send: label) will > help clarify. Any time, data gets queued on the socket > head because the sender is sending faster than tcp can > send (congestion window is small due to slow start or > congestion avoidance or the sender is just plain fast), > the alignment will get broken. Of course this may not be > easily perceived from traces where a single > request/single response at a time model is being followed. > > However, with RDMA/DDP, there may be many messages > outstanding at a time that are not a multiple of segment > size and misalignment will happen unless the sender TCP > is modified. > Let me clarify, it is the sudden loss of alignment that will be rare. The overwhelming majority of unaligned connections will be evidently so very quickly and/or they will only be unaligned for relatively brief portions of their lifespan. A software implementation based upon an existing TCP stack that *attempts* to maintain alignment would be an example of one that will *usually* maintain allignment. One that makes no attempt will quickly be out of alignment, and the receiver will quickly know not to expect alignment on this connection. I would anticipate that most RNICs will be engineered to manage a large number of connetions, when most of those connections are currently aligned. Similarly, most RNICs will be engineered on the assumption that the PMTU is not changing with every packet. It does not mean they are not ready to deal with it when it does happen. >> >> - An RNIC could make a statistical estimate of the >> resources required to handle buffering caused by loss of >> alignment. If those resources are exceeded, it will drop >> the IP datagrams that required the exhausted resources. >> This is no different than what any of the switches, routers >> and other middleboxes between the two RNICs could do if >> *their* statistical estimates are exceeded. Obviously, >> just as with the middleboxes, the statistical estimate >> SHOULD be heavily on the safe side. How safe is a market >> issue, not an IETF issue. >> >> - MPA/TCP is still a reliable transport protocol. It will >> recover from IP datagrams being dropped due to resource >> exhaustion. The validity of the ULP connection will not >> be broken. >> >> - An RNIC also has the option of "slow pathing" connections >> via the host memory itself. Of course this is ultimately >> just another form of buffering, and it can also be exhausted >> so the point has not really changed. But it does emphasize >> that requiring an RNIC to be robust in the face of lost >> alignment DOES NOT require the addition of vast memory >> resources on the RNIC itself. >> >> >> Loss of alignment *will* have a throughput impact. > >I think the issue is whether we will let implementations >decide this or the protocol decide this. I feel we should >let implementation decide this and design a framing >protocol that does not impose this. > Agreed. But as Uri's analysis shows, there are substantial advantages to processing of aligned FPDUs. Only a design optimized for absolute guaranteed performance would not end up working faster when faced with less work. But I agree with the sentiment against it being an imposition. The "will" was meant solely as a prediction, not as a mandate. >> >> Requiring an MPA/TCP receiver to remain robust in the >> event of alignment loss is what is required to truly be >> a reliable IP transport protocol. > > If you look at Jim William's proposal, it provides for a > TCP compliant framing proposal, where the implementor can > decide whether to optimize for aligned senders (like in a > data center), legacy senders (where migration is easier > such as for iSCSI currently) or blend in both. I think > that unless the protocol is targeted at the data center > only, such an approach, which should be acceptable as a > standards track document, is better. > I believe the current MPA draft also accomplishes this, without the addition of a "MPA Hello" sequence or in any way attempting to a priori guarantee alignment on certain connections. Jim's proposal also has a negotiation phase, which is my primary objection. If there is to be a negotiation phase, getting more done with it, as Jim's proposal does, makes the pill slightly less bitter. But my fundamental objection remains that there is no need for a negotiation phase. It does not solve the problems it is alleged to solve, it adds complexity, and has a non-zero cost in terms of network traffic and latency. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 20 22:15:49 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA07620 for ; Thu, 20 Feb 2003 22:15:49 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1L3Mfw06735 for rddp-archive@odin.ietf.org; Thu, 20 Feb 2003 22:22:41 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L3Mep06732 for ; Thu, 20 Feb 2003 22:22:40 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA07612 for ; Thu, 20 Feb 2003 22:15:18 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L3M2p06714; Thu, 20 Feb 2003 22:22:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L3Ljp06672 for ; Thu, 20 Feb 2003 22:21:45 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA07604 for ; Thu, 20 Feb 2003 22:14:22 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 53050249ECA for ; Thu, 20 Feb 2003 22:18:14 -0500 (EST) Date: Thu, 20 Feb 2003 21:19:29 -0600 From: Caitlin Bestler Subject: Re: [rddp] FW: I-D ACTION:draft-williams-iwarp-ift-01.txt To: rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C8F3@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit > 8.2 Overwrite checks > > Before speculative placement is done, the RDDP > implementation MUST insure that no previously placed data > is overwritten. This is necessary to insure that if the > speculative placement is being done in error, that the > error is recoverable. > > This implies that the RDDP implementation must track what > portions of a buffer have been written at any time, or if > the RDDP implementation loses track, then do no further > speculative placements in that buffer. > > If an in-order placement would overwrite a previously > done speculative placement, then that in-order placement > should be done and the preceding speculative placement > regarded as invalid, and needs to be re-done in-order. What is the definition of "previously"? Since the ULP last enabled the use of that memory? Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 20 22:44:52 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA08427 for ; Thu, 20 Feb 2003 22:44:52 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1L3pih08404 for rddp-archive@odin.ietf.org; Thu, 20 Feb 2003 22:51:44 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L3pip08401 for ; Thu, 20 Feb 2003 22:51:44 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA08423 for ; Thu, 20 Feb 2003 22:44:21 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L3p1p08378; Thu, 20 Feb 2003 22:51:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L3oWp08352 for ; Thu, 20 Feb 2003 22:50:32 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA08416 for ; Thu, 20 Feb 2003 22:43:09 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 93E6E249537 for ; Thu, 20 Feb 2003 22:47:02 -0500 (EST) Date: Thu, 20 Feb 2003 21:48:17 -0600 From: Caitlin Bestler Subject: Re: [rddp] FW: I-D ACTION:draft-williams-iwarp-ift-01.txt To: rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit I still prefer the original MPA draft, but the explanations on early placement in Section 8 are excellent. I'd encourage portions to be incorporated into the MPA draft even if no other IFT features are adopted. I have one question. If periodic markers are in use, can they be viewed as a reliable indication of the location of the RDDP PDU (combined with the other fields beig valid)? Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 20 23:12:53 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA08889 for ; Thu, 20 Feb 2003 23:12:53 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1L4JkA10107 for rddp-archive@odin.ietf.org; Thu, 20 Feb 2003 23:19:46 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L4Jkp10104 for ; Thu, 20 Feb 2003 23:19:46 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA08882 for ; Thu, 20 Feb 2003 23:12:22 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L4J5p10086; Thu, 20 Feb 2003 23:19:05 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L4Iap10062 for ; Thu, 20 Feb 2003 23:18:36 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA08848 for ; Thu, 20 Feb 2003 23:11:12 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 8F8D72470AE for ; Thu, 20 Feb 2003 23:15:04 -0500 (EST) Date: Thu, 20 Feb 2003 22:16:19 -0600 From: Caitlin Bestler To: rddp@ietf.org X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] IFT-01 - SP (Speculative Placement) Flag Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit The proposed flags include: > SP Indicates the sender is giving the receiver > permission to do speculative placement. If this bit is > set, the receiver MAY do speculative placement. If not > set, the receiver MUST NOT do speculative placement. See > the chapter below for full description of speculative > placement. Why would the *sender* have an interest in mandating that the receiver not take risks with the *receiver's* memory? Can the sender specify that the receiver not take risks with the data by running certain operating systems? Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 20 23:16:47 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA08932 for ; Thu, 20 Feb 2003 23:16:46 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1L4Nex10291 for rddp-archive@odin.ietf.org; Thu, 20 Feb 2003 23:23:40 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L4Ndp10288 for ; Thu, 20 Feb 2003 23:23:39 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA08925 for ; Thu, 20 Feb 2003 23:16:15 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L4N1p10255; Thu, 20 Feb 2003 23:23:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1L4Mpp10237 for ; Thu, 20 Feb 2003 23:22:51 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id XAA08921 for ; Thu, 20 Feb 2003 23:15:28 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 0195824F807 for ; Thu, 20 Feb 2003 23:19:21 -0500 (EST) Date: Thu, 20 Feb 2003 22:20:35 -0600 From: Caitlin Bestler To: rddp@ietf.org X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] IFT-01 - Two Phase Negotiation. Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit While I am opposed to an MPA/IFT confirmation message in general, I am doubly opposed to requiring it to be a two-phase negotiation. Would it be permissable for ULPs that believed that they had pre-negotiated all details to simply confirm their understanding by having them *both* skip directly to the second phase? Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 21 08:17:40 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA28626 for ; Fri, 21 Feb 2003 08:17:40 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1LDOh022168 for rddp-archive@odin.ietf.org; Fri, 21 Feb 2003 08:24:43 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LDOhp22165 for ; Fri, 21 Feb 2003 08:24:43 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA28608 for ; Fri, 21 Feb 2003 08:17:09 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LDO4p22149; Fri, 21 Feb 2003 08:24:04 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LDNhp22133 for ; Fri, 21 Feb 2003 08:23:43 -0500 Received: from mx01.netapp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA28603 for ; Fri, 21 Feb 2003 08:16:09 -0500 (EST) Received: from frejya.corp.netapp.com (frejya [10.10.20.91]) by mx01.netapp.com (8.12.6/8.12.6/NTAP-1.4) with ESMTP id h1LDJxeZ014738 for ; Fri, 21 Feb 2003 05:19:59 -0800 (PST) Received: from tmt.netapp.com ([10.58.48.88]) by frejya.corp.netapp.com (8.12.6/8.12.6/NTAP-1.4) with ESMTP id h1LDJwM5006858 for ; Fri, 21 Feb 2003 05:19:58 -0800 (PST) Message-Id: <5.2.0.9.2.20030221075355.00ad6040@silver.nane.netapp.com> Date: Fri, 21 Feb 2003 08:16:22 -0500 To: rddp@ietf.org From: "Talpey, Thomas" Subject: Re: [rddp] Reality Check: Avoiding confusion on MPA TCP use In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C8E6@corpmx14.us.dg.co m> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 12:48 PM 2/19/2003, Black_David@emc.com wrote: > The third model is by negotiation of MPA transition by the ULP > using octet stream messages to accomplish a ULP specific > , , three way exchange. >... >In addition, this sort of exchange would also provide a means of >making the current 512-byte fixed marker interval variable, which >is probably going to be needed to seriously deal with unmodified >TCP stacks (e.g., are 16 syscalls or other send events/descriptors >really reasonable to send an 8k DDP data block?). I've been resisting the urge to complete the proposal I made on "tunable marker intervals", but I do want to inject something here. It's not necessary to exchange a 3-way hello handshake to provide for maker tuning. Because the markers in each direction are independent, and are processed only by the receiver, it is trivially easy to simply exchange them without any negotiation or acknowledgement. In a nutshell, each peer would initiate its MPA level exchange with a marker containing the interval required by its receiver. If 0, no markers are to be inserted. Sends can proceed in either direction immediately upon receipt of the initial marker. I believe the upper layer which invoked MPA is responsible for ensuring its recognition by the peer, and not its designers having to engage in exploration whether the pattern of all-0's, all-1's or the ascii string "MPA!" is adequate protection. Tom. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 21 10:24:12 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA03299 for ; Fri, 21 Feb 2003 10:24:12 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1LFVI130513 for rddp-archive@odin.ietf.org; Fri, 21 Feb 2003 10:31:18 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LFVIp30510 for ; Fri, 21 Feb 2003 10:31:18 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA03285 for ; Fri, 21 Feb 2003 10:23:41 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LFU9p30448; Fri, 21 Feb 2003 10:30:09 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LFTlp30391 for ; Fri, 21 Feb 2003 10:29:47 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA03231 for ; Fri, 21 Feb 2003 10:22:10 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 3B7CC245C32; Fri, 21 Feb 2003 10:26:02 -0500 (EST) Date: Fri, 21 Feb 2003 09:27:18 -0600 From: Caitlin Bestler Subject: Re: [rddp] Reality Check: Avoiding confusion on MPA TCP use To: "Talpey, Thomas" , rddp@ietf.org X-Priority: 3 In-Reply-To: <5.2.0.9.2.20030221075355.00ad6040@silver.nane.netapp.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/21/03, Talpey, Thomas wrote: >At 12:48 PM 2/19/2003, Black_David@emc.com wrote: >> The third model is by negotiation of MPA transition by the ULP >> using octet stream messages to accomplish a ULP specific >> , , three way exchange. >>... >>In addition, this sort of exchange would also provide a means of >>making the current 512-byte fixed marker interval variable, which >>is probably going to be needed to seriously deal with unmodified >>TCP stacks (e.g., are 16 syscalls or other send events/descriptors >>really reasonable to send an 8k DDP data block?). > >I've been resisting the urge to complete the proposal I >made on "tunable marker intervals", but I do want to >inject something here. > >It's not necessary to exchange a 3-way hello handshake to >provide for maker tuning. Because the markers in each >direction are independent, and are processed only by the >receiver, it is trivially easy to simply exchange them >without any negotiation or acknowledgement. > >In a nutshell, each peer would initiate its MPA level >exchange with a marker containing the interval required by >its receiver. If 0, no markers are to be inserted. Sends >can proceed in either direction immediately upon receipt >of the initial marker. > >I believe the upper layer which invoked MPA is responsible >for ensuring its recognition by the peer, and not its >designers having to engage in exploration whether the >pattern of all-0's, all-1's or the ascii string "MPA!" is >adequate protection. > I'd go even farther. The marker interval, like all other protocol details, is a shared assumption. The ULP peers have an amazing wide variety of ways to establish that shared assumption: a pre-MPA mode handshake, system configuration, shared header files and/or protocol specifications. If this assumption was in error, it will quickly be determined whether or not there is "confirmation" message. And there are already sufficient safeguards to guarantee that only the ULP in question (who is a party to the faulty assumption) is at risk. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 21 10:52:38 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA04268 for ; Fri, 21 Feb 2003 10:52:38 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1LFxj632693 for rddp-archive@odin.ietf.org; Fri, 21 Feb 2003 10:59:45 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LFxjp32690 for ; Fri, 21 Feb 2003 10:59:45 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA04219 for ; Fri, 21 Feb 2003 10:52:07 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LFx5p32655; Fri, 21 Feb 2003 10:59:05 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LFwmp32589 for ; Fri, 21 Feb 2003 10:58:48 -0500 Received: from mxic1.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA04185 for ; Fri, 21 Feb 2003 10:51:10 -0500 (EST) From: Black_David@emc.com Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1KDZAR6Y>; Fri, 21 Feb 2003 10:54:59 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C902@corpmx14.us.dg.com> To: cait@asomi.com, rddp@ietf.org Subject: RE: [rddp] IFT-01 - Two Phase Negotiation. Date: Fri, 21 Feb 2003 10:52:28 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > While I am opposed to an MPA/IFT confirmation message in > general, I am doubly opposed to requiring it to be a > two-phase negotiation. > > Would it be permissable for ULPs that believed that they > had pre-negotiated all details to simply confirm their > understanding by having them *both* skip directly to > the second phase? As I understand "phases" in the IFT draft, each "phase" is one-way: PH Adoption is two phase negotiation, with phases numbered 0, and 1. Phase 0 is sent from node A to node B. On receipt of phase 0, node B sends phase 1 back to node A. This bit indicates phase 0 or 1. So, the request to "skip directly to the second phase" is a request to send only the B to A adaption indication, and not the A to B one. That seems questionable - is this about optimizing the size of the first TCP segment in only one of the two directions? I think the exchange of adaption indications to confirm the ULP understanding is still needed. I don't see much of a gain in allowing the adaption indications to be exchanged independently (i.e., responder does not wait for the initial indication to arrive before sending its indication) as with a little care, the initial adaption indication should come right behind the ACK that completes TCP's 3-way handshake. Having the second adaption indication respond to the first one (rather than be independent of it) allows the receiver to tell the sender *exactly* what's wrong when things don't work, which can help ensure that a retry will work. OTOH, I don't see a problem with both sides sending traffic immediately behind their adaption indications. An issue that arises is that the initial sender has to guess or have other knowledge as to whether to use markers and/or what marker interval to use. If the guess is wrong, the connection probably has to be abandoned, but the returned adaption indication should indicate what will work on retry. As long as there's language about how the receiver is supposed to deal with this "sender guessed wrong" case, I don't see a problem. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 21 12:04:29 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA07074 for ; Fri, 21 Feb 2003 12:04:29 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1LHBbJ05827 for rddp-archive@odin.ietf.org; Fri, 21 Feb 2003 12:11:37 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LHBbp05824 for ; Fri, 21 Feb 2003 12:11:37 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA07055 for ; Fri, 21 Feb 2003 12:03:57 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LHAPp05736; Fri, 21 Feb 2003 12:10:25 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LH3Qp04349 for ; Fri, 21 Feb 2003 12:03:26 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA06724 for ; Fri, 21 Feb 2003 11:55:46 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id AC3DF245ED2; Fri, 21 Feb 2003 11:59:38 -0500 (EST) Date: Fri, 21 Feb 2003 11:00:55 -0600 From: Caitlin Bestler Subject: RE: [rddp] IFT-01 - Two Phase Negotiation. To: Black_David@emc.com, rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C902@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/21/03, Black_David@emc.com wrote: >> While I am opposed to an MPA/IFT confirmation message in >> general, I am doubly opposed to requiring it to be a >> two-phase negotiation. >> >> Would it be permissable for ULPs that believed that they >> had pre-negotiated all details to simply confirm their >> understanding by having them *both* skip directly to >> the second phase? > >As I understand "phases" in the IFT draft, each "phase" >is one-way: > >PH Adoption is two phase negotiation, with phases > numbered 0, and 1. Phase 0 is sent from node A > to node B. On receipt of phase 0, node B sends phase > 1 back to node A. This bit indicates phase 0 or 1. > >So, the request to "skip directly to the second phase" is >a request to send only the B to A adaption indication, and >not the A to B one. That seems questionable - is this >about optimizing the size of the first TCP segment in only >one of the two directions? > Let me repeat my earlier caveat: I don't believe MPA-mode confirmation messages are required at all. I believe an MPA-mode confirmation message will waste a TCP segment, and it solves no real problem. Even a single TCP segment wasted is unacceptable if it does not provide anything of real value. Further, on long-haul connections one TCP segment while the connection is still in its slow start is not insignifigant. That said, a negotiation is worse. It is not only an additional TCP segment, it is an extra wait state. As drafted the responding MPA must *wait* for the phase 0 MPA-config message to arrive. That is a round-trip delay. It is an extra state. The MPA engine must be capable of *generating* messages, not merely wrapping messages given to it. The result is far uglier than just the addition of an MPA-Mode confirmation message. If each side can merely announce its configuration, based on whatever pre-MPA negotiations that may have taken place (see my reply to Tom Talpey), then no additional states have been created. The unjustified damage is limited to the wasted TCP segment. IHow does it truly made sense for MPA/IFT layer characteristics to be dynamically determined on a per connection basis in a manner transparent to the ULP? It seems to me that the existence of a large number of protocol options is not an argument for a mode-initializing packet, rather it is an argument to eliminate those options (or to make them mandatory). Most of the options discussed, if needed at all, would properly be decided by a ULP profile. I have seen no justification for establishing any of them on a per-connection basis. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 21 12:20:39 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA07590 for ; Fri, 21 Feb 2003 12:20:39 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1LHRmx06498 for rddp-archive@odin.ietf.org; Fri, 21 Feb 2003 12:27:48 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LHRmp06495 for ; Fri, 21 Feb 2003 12:27:48 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA07572 for ; Fri, 21 Feb 2003 12:20:08 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LHQ3p06436; Fri, 21 Feb 2003 12:26:03 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LHP6p06380 for ; Fri, 21 Feb 2003 12:25:06 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA07441 for ; Fri, 21 Feb 2003 12:17:26 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRKS838>; Fri, 21 Feb 2003 12:21:17 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C904@corpmx14.us.dg.com> To: cait@asomi.com, rddp@ietf.org Subject: RE: [rddp] IFT-01 - Two Phase Negotiation. Date: Fri, 21 Feb 2003 12:18:35 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > That said, a negotiation is worse. It is not only an > additional TCP segment, it is an extra wait state. As > drafted the responding MPA must *wait* for the phase 0 > MPA-config message to arrive. That is a round-trip delay. Yes, there's a "wait," but it's not a round-trip delay if the MPA-config message is back-to-back with the TCP ACK that completes the 3-way TCP handshake. Remember that the passive side of the TCP connection setup has to wait for that TCP ACK before it can send. > If each side can merely announce its configuration, based on > whatever pre-MPA negotiations that may have taken place (see > my reply to Tom Talpey), then no additional states have been > created. The unjustified damage is limited to the wasted TCP > segment. That's an acceptable approach, but the difference in "damage" is small between whether the passive side blindly sends the configuration announcement vs. looks at the MPA-config message that is usually immediately behind the ACK that it has to wait for anyway. There may be a gain in error-case behavior (if the response depends on the input, the response can say "do this" (or "go away"), vs. relying on both sides to draw the same conclusion about what will work from the exchanged configurations. There are delay and computation costs, but a round-trip delay is definitely *not* involved. Either design approach is acceptable. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Friday, February 21, 2003 12:01 PM > To: Black_David@emc.com; rddp@ietf.org > Subject: RE: [rddp] IFT-01 - Two Phase Negotiation. > > > On 2/21/03, Black_David@emc.com wrote: > > >> While I am opposed to an MPA/IFT confirmation message in > >> general, I am doubly opposed to requiring it to be a > >> two-phase negotiation. > >> > >> Would it be permissable for ULPs that believed that they > >> had pre-negotiated all details to simply confirm their > >> understanding by having them *both* skip directly to > >> the second phase? > > > >As I understand "phases" in the IFT draft, each "phase" > >is one-way: > > > >PH Adoption is two phase negotiation, with phases > > numbered 0, and 1. Phase 0 is sent from node A > > to node B. On receipt of phase 0, node B sends > phase > > 1 back to node A. This bit indicates phase 0 > or 1. > > > >So, the request to "skip directly to the second phase" is > >a request to send only the B to A adaption indication, and > >not the A to B one. That seems questionable - is this > >about optimizing the size of the first TCP segment in only > >one of the two directions? > > > > Let me repeat my earlier caveat: I don't believe MPA-mode > confirmation messages are required at all. I believe an > MPA-mode confirmation message will waste a TCP segment, > and it solves no real problem. Even a single TCP segment > wasted is unacceptable if it does not provide anything of > real value. Further, on long-haul connections one TCP > segment while the connection is still in its slow start > is not insignifigant. > > That said, a negotiation is worse. It is not only an > additional TCP segment, it is an extra wait state. As > drafted the responding MPA must *wait* for the phase 0 > MPA-config message to arrive. That is a round-trip delay. > It is an extra state. The MPA engine must be capable of > *generating* messages, not merely wrapping messages given to > it. The result is far uglier than just the addition of an > MPA-Mode confirmation message. > > If each side can merely announce its configuration, based on > whatever pre-MPA negotiations that may have taken place (see > my reply to Tom Talpey), then no additional states have been > created. The unjustified damage is limited to the wasted TCP > segment. > > IHow does it truly made sense for MPA/IFT layer > characteristics to be dynamically determined on a per > connection basis in a manner transparent to the ULP? > > It seems to me that the existence of a large number of > protocol options is not an argument for a mode-initializing > packet, rather it is an argument to eliminate those options > (or to make them mandatory). > > Most of the options discussed, if needed at all, would > properly be decided by a ULP profile. I have seen no > justification for establishing any of them on a > per-connection basis. > > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 21 14:08:31 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA11518 for ; Fri, 21 Feb 2003 14:08:31 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1LJFhQ13824 for rddp-archive@odin.ietf.org; Fri, 21 Feb 2003 14:15:43 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LJFhp13821 for ; Fri, 21 Feb 2003 14:15:43 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA11497 for ; Fri, 21 Feb 2003 14:08:00 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LJELp13779; Fri, 21 Feb 2003 14:14:21 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LJDDp13658 for ; Fri, 21 Feb 2003 14:13:13 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA11398 for ; Fri, 21 Feb 2003 14:05:30 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 132822458F5; Fri, 21 Feb 2003 14:09:19 -0500 (EST) Date: Fri, 21 Feb 2003 13:10:35 -0600 From: Caitlin Bestler Subject: RE: [rddp] IFT-01 - Two Phase Negotiation. To: Black_David@emc.com, rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C904@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/21/03, Black_David@emc.com wrote: >That's an acceptable approach, but the difference in >"damage" is small between whether the passive side blindly >sends the configuration announcement vs. looks at the >MPA-config message that is usually immediately behind the >ACK that it has to wait for anyway. There may be a gain >in error-case behavior (if the response depends on the >input, the response can say "do this" (or "go away"), vs. >relying on both sides to draw the same conclusion about >what will work from the exchanged configurations. There >are delay and computation costs, but a round-trip delay is >definitely *not* involved. > >Either design approach is acceptable. I would agree that the extra delay can be avoided if the passive side is allowed to send its confirmation message on a speculative basis. If the other side's message does not confirm those assumptions the connection MUST be aborted. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 21 14:53:51 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA12695 for ; Fri, 21 Feb 2003 14:53:51 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1LK12e16383 for rddp-archive@odin.ietf.org; Fri, 21 Feb 2003 15:01:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LK12p16380 for ; Fri, 21 Feb 2003 15:01:02 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA12685 for ; Fri, 21 Feb 2003 14:53:19 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LJx2p16286; Fri, 21 Feb 2003 14:59:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LJwDp16259 for ; Fri, 21 Feb 2003 14:58:13 -0500 Received: from mx01.netapp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA12659 for ; Fri, 21 Feb 2003 14:50:30 -0500 (EST) Received: from frejya.corp.netapp.com (frejya [10.10.20.91]) by mx01.netapp.com (8.12.6/8.12.6/NTAP-1.4) with ESMTP id h1LJsKeZ010432 for ; Fri, 21 Feb 2003 11:54:20 -0800 (PST) Received: from tmt.netapp.com (tmt [10.97.1.39]) by frejya.corp.netapp.com (8.12.6/8.12.6/NTAP-1.4) with ESMTP id h1LJsJM5000732; Fri, 21 Feb 2003 11:54:19 -0800 (PST) Message-Id: <5.2.0.9.2.20030221144545.0398fd88@silver.nane.netapp.com> Date: Fri, 21 Feb 2003 14:54:07 -0500 To: rddp@ietf.org From: "Talpey, Thomas" Subject: RE: [rddp] IFT-01 - Two Phase Negotiation. In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C904@corpmx14.us.dg.co m> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 12:18 PM 2/21/2003, Black_David@emc.com wrote: >> That said, a negotiation is worse. It is not only an >> additional TCP segment, it is an extra wait state. As >> drafted the responding MPA must *wait* for the phase 0 >> MPA-config message to arrive. That is a round-trip delay. > >Yes, there's a "wait," but it's not a round-trip delay >if the MPA-config message is back-to-back with the TCP >ACK that completes the 3-way TCP handshake. Remember >that the passive side of the TCP connection setup has >to wait for that TCP ACK before it can send. I don't see why the passive TCP must await the ACK of its SYN. It's perfectly acceptable for TCPs to include data with their initial handshake, on both sides. But more to the point, what of the upper layer protocols, which must also exchange messages? If framing handshake must complete, then the initial upper layer handshake is delayed for a full roundtrip. Imposing additional handshakes in the lower layers will certainly impact availability. Tom. > >> If each side can merely announce its configuration, based on >> whatever pre-MPA negotiations that may have taken place (see >> my reply to Tom Talpey), then no additional states have been >> created. The unjustified damage is limited to the wasted TCP >> segment. > >That's an acceptable approach, but the difference in "damage" >is small between whether the passive side blindly sends >the configuration announcement vs. looks at the MPA-config >message that is usually immediately behind the ACK that it >has to wait for anyway. There may be a gain in error-case >behavior (if the response depends on the input, the response >can say "do this" (or "go away"), vs. relying on both sides >to draw the same conclusion about what will work from the >exchanged configurations. There are delay and computation >costs, but a round-trip delay is definitely *not* involved. > >Either design approach is acceptable. > >Thanks, >--David >---------------------------------------------------- >David L. Black, Senior Technologist >EMC Corporation, 176 South St., Hopkinton, MA 01748 >+1 (508) 293-7953 FAX: +1 (508) 293-7786 >black_david@emc.com Mobile: +1 (978) 394-7754 >---------------------------------------------------- > >> -----Original Message----- >> From: Caitlin Bestler [mailto:cait@asomi.com] >> Sent: Friday, February 21, 2003 12:01 PM >> To: Black_David@emc.com; rddp@ietf.org >> Subject: RE: [rddp] IFT-01 - Two Phase Negotiation. >> >> >> On 2/21/03, Black_David@emc.com wrote: >> >> >> While I am opposed to an MPA/IFT confirmation message in >> >> general, I am doubly opposed to requiring it to be a >> >> two-phase negotiation. >> >> >> >> Would it be permissable for ULPs that believed that they >> >> had pre-negotiated all details to simply confirm their >> >> understanding by having them *both* skip directly to >> >> the second phase? >> > >> >As I understand "phases" in the IFT draft, each "phase" >> >is one-way: >> > >> >PH Adoption is two phase negotiation, with phases >> > numbered 0, and 1. Phase 0 is sent from node A >> > to node B. On receipt of phase 0, node B sends >> phase >> > 1 back to node A. This bit indicates phase 0 >> or 1. >> > >> >So, the request to "skip directly to the second phase" is >> >a request to send only the B to A adaption indication, and >> >not the A to B one. That seems questionable - is this >> >about optimizing the size of the first TCP segment in only >> >one of the two directions? >> > >> >> Let me repeat my earlier caveat: I don't believe MPA-mode >> confirmation messages are required at all. I believe an >> MPA-mode confirmation message will waste a TCP segment, >> and it solves no real problem. Even a single TCP segment >> wasted is unacceptable if it does not provide anything of >> real value. Further, on long-haul connections one TCP >> segment while the connection is still in its slow start >> is not insignifigant. >> >> That said, a negotiation is worse. It is not only an >> additional TCP segment, it is an extra wait state. As >> drafted the responding MPA must *wait* for the phase 0 >> MPA-config message to arrive. That is a round-trip delay. >> It is an extra state. The MPA engine must be capable of >> *generating* messages, not merely wrapping messages given to >> it. The result is far uglier than just the addition of an >> MPA-Mode confirmation message. >> >> If each side can merely announce its configuration, based on >> whatever pre-MPA negotiations that may have taken place (see >> my reply to Tom Talpey), then no additional states have been >> created. The unjustified damage is limited to the wasted TCP >> segment. >> >> IHow does it truly made sense for MPA/IFT layer >> characteristics to be dynamically determined on a per >> connection basis in a manner transparent to the ULP? >> >> It seems to me that the existence of a large number of >> protocol options is not an argument for a mode-initializing >> packet, rather it is an argument to eliminate those options >> (or to make them mandatory). >> >> Most of the options discussed, if needed at all, would >> properly be decided by a ULP profile. I have seen no >> justification for establishing any of them on a >> per-connection basis. >> >> >> Caitlin Bestler >> http://asomi.com/CaitlinBestler/ >> >_______________________________________________ >rddp mailing list >rddp@ietf.org >https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 21 15:25:41 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA13635 for ; Fri, 21 Feb 2003 15:25:41 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1LKWro18454 for rddp-archive@odin.ietf.org; Fri, 21 Feb 2003 15:32:53 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LKWqp18451 for ; Fri, 21 Feb 2003 15:32:52 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA13611 for ; Fri, 21 Feb 2003 15:25:09 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LKW8p18436; Fri, 21 Feb 2003 15:32:08 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LKV8p18413 for ; Fri, 21 Feb 2003 15:31:08 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA13574 for ; Fri, 21 Feb 2003 15:23:25 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRKTNFW>; Fri, 21 Feb 2003 15:27:16 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C909@corpmx14.us.dg.com> To: Thomas.Talpey@netapp.com, rddp@ietf.org Subject: RE: [rddp] IFT-01 - Two Phase Negotiation. Date: Fri, 21 Feb 2003 15:24:46 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > >Yes, there's a "wait," but it's not a round-trip delay > >if the MPA-config message is back-to-back with the TCP > >ACK that completes the 3-way TCP handshake. Remember > >that the passive side of the TCP connection setup has > >to wait for that TCP ACK before it can send. > > I don't see why the passive TCP must await the ACK of its > SYN. It's perfectly acceptable for TCPs to include data with > their initial handshake, on both sides. I stand corrected, but this doesn't help as much as it might, as RFC 793 says: Several examples of connection initiation follow. Although these examples do not show connection synchronization using data-carrying segments, this is perfectly legitimate, so long as the receiving TCP doesn't deliver the data to the user until it is clear the data is valid (i.e., the data must be buffered at the receiver until the connection reaches the ESTABLISHED state). So, while the adaption indications could be sent in the TCP handshake segments, one can't actually do anything with them until the TCP handshake is complete. > But more to the point, what of the upper layer protocols, > which must also exchange messages? If framing handshake > must complete, then the initial upper layer handshake is > delayed for a full roundtrip. Imposing additional handshakes > in the lower layers will certainly impact availability. It doesn't have to complete before data is sent - as I've said earlier, I have no problem with sending framed data immediately after the adaption indications. The only minor drawback is that if arrival of the second message at the sender tells the sender that the data it sent previously wasn't formatted in a fashion the receiver could deal with, the connection is probably unrecoverable at that point and likely needs to be terminated. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 21 16:33:56 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15622 for ; Fri, 21 Feb 2003 16:33:56 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1LLfAK23144 for rddp-archive@odin.ietf.org; Fri, 21 Feb 2003 16:41:10 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LLfAp23141 for ; Fri, 21 Feb 2003 16:41:10 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15614 for ; Fri, 21 Feb 2003 16:33:25 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LLdAp23033; Fri, 21 Feb 2003 16:39:10 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1LLcYp22997 for ; Fri, 21 Feb 2003 16:38:34 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15496 for ; Fri, 21 Feb 2003 16:30:49 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 18B682467F3 for ; Fri, 21 Feb 2003 16:34:42 -0500 (EST) Date: Fri, 21 Feb 2003 15:35:58 -0600 From: Caitlin Bestler To: rddp@ietf.org X-Priority: 3 Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Subject: [rddp] Applicability draft Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit I've submitted a first draft at an applicability statement. Until it shows up officially it can be fetched at http://asomi.com/draft-bestler-rddp-applicability-00.txt Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 21 19:18:33 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA19325 for ; Fri, 21 Feb 2003 19:18:33 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1M0Pp500368 for rddp-archive@odin.ietf.org; Fri, 21 Feb 2003 19:25:51 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1M0Ppp00365 for ; Fri, 21 Feb 2003 19:25:51 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA19310 for ; Fri, 21 Feb 2003 19:18:01 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1M0P4p00348; Fri, 21 Feb 2003 19:25:04 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1M0Osp00324 for ; Fri, 21 Feb 2003 19:24:54 -0500 Received: from dmz1.silverbacksystems.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA19303 for ; Fri, 21 Feb 2003 19:17:05 -0500 (EST) Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id 524C5D1BC; Fri, 21 Feb 2003 16:20:44 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id 159BF19080; Fri, 21 Feb 2003 16:20:44 -0800 (PST) Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii" Date: Fri, 21 Feb 2003 16:20:29 -0800 From: "Somesh Gupta" Importance: Normal In-Reply-To: Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 15721-4E681DCB; Fri, 21 Feb 2003 16:20:17 -0800 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id 375E11B07; Fri, 21 Feb 2003 16:20:17 -0800 (PST) Subject: RE: [rddp] MPA TCP analysis draft To: "Caitlin Bestler" , X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit I wanted to explore the assumption behind the memory sizes somewhat more. The formulae given are BufferSizeNAF = K1 * EMSS * number_of_connections + K2 * EMSS and BufferSizeAF = K2 * EMSS There is no question that overall there is a lot of simplification if there is header alignment, and there is only upper layer PDU per TCP segment (BTW by having multiple Upper Layer PDUs per segment, a lot of the simplification has been lost). However, it is important that the protocol not be optimized only around this, since alignment cannot be maintained by software based senders without modifications to the TCP stack. I pointed out references to the BSD TCP software in one of my previous emails, and can repeat it if needed. Now to discussing the formuale: 1. There is an assumption that the DDP & RDMA headers are not needed or buffered elsewhere? For an implementation that wants to check for and protect against violation of the DDP/RDMA protocols, the headers will be needed later when the gaps have been filled in (an example is that untagged MSN must be in increasing order). The oft repeated rule is out of order placement and not out of order delivery. We can make a number of assumptions here. a. It is not necessary as we don't expect this to happen. b. Our NIC is only a TCP Direct Data Placement NIC and defers all DDP/RDMA processing to the host c. We have to buffer the headers. In such a case, the BufferSize becomes a function of the number of connections regardless of the method chosen. (see discussion of number of connections below) 2. There is an assumption of markers being smaller than the EMSS. Since markers are essential to locating the header (either in MPA or in IFT), you have to first find a marker, and only then can you place the data if segments are out of order. As Divid pointed out, a smaller marker interval puts an unacceptable burden on a software client (I thing the example was of a 64KB read/write requiring 140+ markers - each of which causes creation of that many segments, plus creating and inserting markers and marker vectors etc) - worse than simply using a NIC doing a large TCP send or TOE offload. a. Either severely penalize software based senders (at hopefully your own risk rather than institutionalizing the risk via the protocol). b. or alternatively add a factor of K1 * Marker Interval/EMSS * number_of_connections to the buffering needs. 3. The number of connections in the above equations is really the number of connections with out of order segments - and not the total number of connections. Of course, for this you would have to have the separate header and data CRC as proposed by Jim. Hopefully this is a much smaller number than the number of connections. Of course, if every other segment is out of order, one does not have worry about buffering or memory speeds etc because TCP algorithms will force you to a rate in the range of KBytes and not Gbits. TCP is not a credit based mechanism but uses the AIMD algorithms to "detect" network congestion. It will assume that it needs to back off significantly to prevent a congestion collapse of the network. Somesh > -----Original Message----- > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > Caitlin Bestler > Sent: Tuesday, February 18, 2003 1:55 PM > To: rddp@ietf.org > Subject: Re: [rddp] MPA TCP analysis draft > > > The MPA analysis draft offers the following formula starting > on page 10: > > > For the case when the FPDU is not aligned with the TCP > > segment, it may take, on average, 2 TCP segments to > > assemble one FPDU. Therefore, the receiver has to > > allocate BufferSizeNAF (Buffer Size, Non-Aligned FPDU) > > octets: > > > > BufferSizeNAF = K1* EMSS * number_of_connections + > > K2 * EMSS > > > > > > Where K1 and K2 are implementation dependent constants > > and EMSS is the effective maximum segment size. > > > > For example, a 1 Gbps link with 10,000 connections and an > > EMSS of 1500B would require 15 MB of memory. Often the > > number of connections used scales with the network speed, > > aggravating the situation for higher speeds. > > > The derivation of this formula does not seem to be supported > anywhere in the text. > > In fact I can't figure out where it came from, or what > assumptions are being swept under the rug in K1 and K2. > > As pointed out, 2 adjacent TCP segments should allow the > assembly of one FPDU. (In fact it is only the strangest > of corner cases where this is not true). So the question > becomes, how many TCP segments might have to be buffered > before an FPDU is assembled and can be placed? > > Phrased that way, the answer is obviously half of the > TCP segments that can be in flight. The worst case scenario > is clearly dropping every other TCP segment. (And as > previously noted, if a load-balancing router dispatches > successive segments by alternate path that is *exactly* what > could happen if one path is delayed). > > I believe that is a simpler explanation, and formula. > It does not change any of the conclusions, however. > > If the formula presented actually represent a *lower* > estimate of the required buffering then the rationale > should be explained. > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 21 21:22:30 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA21432 for ; Fri, 21 Feb 2003 21:22:30 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1M2Tn306566 for rddp-archive@odin.ietf.org; Fri, 21 Feb 2003 21:29:49 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1M2Tnp06563 for ; Fri, 21 Feb 2003 21:29:49 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA21398 for ; Fri, 21 Feb 2003 21:21:59 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1M2T9p06547; Fri, 21 Feb 2003 21:29:09 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1M2S6p06522 for ; Fri, 21 Feb 2003 21:28:06 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA21376 for ; Fri, 21 Feb 2003 21:20:16 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 9994C24511E; Fri, 21 Feb 2003 21:24:06 -0500 (EST) Date: Fri, 21 Feb 2003 20:25:26 -0600 From: Caitlin Bestler Subject: RE: [rddp] MPA TCP analysis draft To: Somesh Gupta , rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/21/03, Somesh Gupta wrote: > >We can make a number of assumptions here. > >a. It is not necessary as we don't expect this to happen. >b. Our NIC is only a TCP Direct Data Placement NIC and >defers > all DDP/RDMA processing to the host >c. We have to buffer the headers. In such a case, the > BufferSize becomes a function of the number of > connections regardless of the method chosen. (see > discussion of number of connections below) > My working assumption is that placement is done by the RNIC. The data that controls the placement may be edited on the RNIC based on queued requests, or directly edited by host based software. The RDMA Read Engine could be implemented either as another block on the RNIC or on the host. There is no need to buffer received segments after placement. >2. There is an assumption of markers being smaller than >the EMSS. Since markers are essential to locating the >header (either in MPA or in IFT), you have to first find a >marker, and only then can you place the data if segments >are out of order. As David pointed out, a smaller marker >interval puts an unacceptable burden on a software client >(I thing the example was of a 64KB read/write requiring >140+ markers - each of which causes creation of that many >segments, plus creating and inserting markers and marker >vectors etc) - worse than simply using a NIC doing a large >TCP send or TOE offload. > The pattern where the true payload is a repeating pattern. 140 markers does not require 140 distinct descriptors. If you look up old mini-computers you'll find many examples of repeating groups in DMA control. > >Of course, if every other segment is out of order, one >does not have worry about buffering or memory speeds etc >because TCP algorithms will force you to a rate in the >range of KBytes and not Gbits. TCP is not a credit based >mechanism but uses the AIMD algorithms to "detect" network >congestion. It will assume that it needs to back off >significantly to prevent a congestion collapse of the >network. > The primary scenario that I can come up with is when one of two load balanced paths fails. That will indeed result in a rapid drop in the memory speed, but that won't help you *this* msec. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sat Feb 22 00:07:23 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA23994 for ; Sat, 22 Feb 2003 00:07:23 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1M5El114474 for rddp-archive@odin.ietf.org; Sat, 22 Feb 2003 00:14:47 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1M5Ekp14471 for ; Sat, 22 Feb 2003 00:14:46 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA23965 for ; Sat, 22 Feb 2003 00:06:52 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1M5E6p14459; Sat, 22 Feb 2003 00:14:06 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1M5DRp14437 for ; Sat, 22 Feb 2003 00:13:27 -0500 Received: from sandmail.sandburst.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id AAA23962 for ; Sat, 22 Feb 2003 00:05:33 -0500 (EST) To: "Williams, Jim" Cc: "'rddp@ietf.org'" Subject: Re: [rddp] STag Invalidation Question In-Reply-To: Message from "Williams, Jim" of "Fri, 07 Feb 2003 11:01:08 EST." <3356669BBE90C448AD4645C843E2BF289B8FE1@xbl.ma.emulex.com> References: <3356669BBE90C448AD4645C843E2BF289B8FE1@xbl.ma.emulex.com> Date: Sat, 22 Feb 2003 00:09:24 -0500 From: Stephen Bailey Message-Id: <20030222050923.D83F64EFB@sandmail.sandburst.com> Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Jim, > PROPOSAL: STags are always invalidated immediately on > on completion of RDDP writes. This is done > before delivery of the data to the application. > Therefore all tagged buffers are "write once" > and must be re-registered before they can > be written again. Others have pointed out the advantages of NOT adopting this model, but I feel the need to further amplify their point. Arbitrary reuse of mapped buffer is an extremely powerful mechanism for building high performance ULPs. RDDP should absolutely permit arbitrary buffer reuse. A concern is that this extreme power can result in `unsafe' ULPs (and mappings). That is true. The right way to view this is that different ULPs have different safety and performance requirements. Applications that need utmost performance and are `closed system' can employ less safe techiques because the environment is controlled and the implementations are tested to a level associated with the risk. Frequently these are applications that will never be standardized anywhere---they come from programmers just sitting down and building on top of supplied communication services. Obviously, somebody writing, say, a database application that is going to handle zillions of dollars is going to do a little testing, and knows best about how to get complete application safety. On the other hand, applications that are more `open system', like Internet data transport, with many interoperating implementations, should be careful and avail themselves of as safe a mapping to RDDP as possible. Single use STags are one of the tools to do this. This discussion recurs frequently enough that I think once the protocol documents have settled and we've got some miles on the implementations, we (somebody) should probably write a BCP on how to do safe/defensive ULP mapping to RDDP, and some guidelines on when to strive for safety in a ULP mapping/application. Steph _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sat Feb 22 12:59:30 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA13854 for ; Sat, 22 Feb 2003 12:59:30 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1MI79h29718 for rddp-archive@odin.ietf.org; Sat, 22 Feb 2003 13:07:09 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1MI79p29707 for ; Sat, 22 Feb 2003 13:07:09 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA13851 for ; Sat, 22 Feb 2003 12:58:59 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1MI6Mp29375; Sat, 22 Feb 2003 13:06:22 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1MI5ep29357 for ; Sat, 22 Feb 2003 13:05:40 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA13829 for ; Sat, 22 Feb 2003 12:57:30 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 9D7E4244BC2; Sat, 22 Feb 2003 13:01:20 -0500 (EST) Date: Sat, 22 Feb 2003 12:02:42 -0600 From: Caitlin Bestler Subject: RE: [rddp] MPA TCP analysis draft To: Somesh Gupta , rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/21/03, Somesh Gupta wrote: >1. There is an assumption that the DDP & RDMA headers are >not needed or buffered elsewhere? For an implementation >that wants to check for and protect against violation of >the DDP/RDMA protocols, the headers will be needed later >when the gaps have been filled in (an example is that >untagged MSN must be in increasing order). The oft >repeated rule is out of order placement and not out of >order delivery. I presume you are talking about something like receiving two DDP segments which are back-to-back in LLP ordering for the DDP stream, but claim to finish MSN: N and MSN: N+2. The receiving DDP MAY detect that MSN N+1 can never arrive and therefore the stream is in error and should be terminated. I would suggest that it SHOULD have better uses for its processing capacity. Rules requiring MSNs to be in sequence are intended to allow the receiver to optimize its code, not to add to a list of validity checks that it MUST perform. So if the receiving DDP notes that it cannot properly generate a completion to the ULP because there is a gap in the MSNs, it should definitely terminate the connection and report an error. But that's detecting the error as a by-product of its normal operations. Similarly, A duplicate MSN should be detected, because the first time the final segment of MSN N is received it should result in completion of the pre-posted receive operation. The second one would have nothing to complete, hence BOING. The receiver is only obligated to validate its peer run-time compliance with the spec when out-of-spec behaviour could be used to compromise system security. But Out-of-sequence MSNs do not place the receiver's resources at risk. It cannot be used to obtain access to any ULP buffers that could not have been accessed in order. LLP checks against improper sequencing are common because they are a method of spotting forged packets. But I would submit that this is an LLP responsibility, DDP should presume that the packet passed to it is a valid part of the DDP stream and not attempt to do the LLP's job. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sat Feb 22 13:06:05 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA13946 for ; Sat, 22 Feb 2003 13:06:05 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1MIDiT30272 for rddp-archive@odin.ietf.org; Sat, 22 Feb 2003 13:13:44 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1MIDip30269 for ; Sat, 22 Feb 2003 13:13:44 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA13943 for ; Sat, 22 Feb 2003 13:05:34 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1MID1p30243; Sat, 22 Feb 2003 13:13:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1MICLp30218 for ; Sat, 22 Feb 2003 13:12:22 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA13931 for ; Sat, 22 Feb 2003 13:04:11 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 66976244A63; Sat, 22 Feb 2003 13:08:03 -0500 (EST) Date: Sat, 22 Feb 2003 12:09:24 -0600 From: Caitlin Bestler Subject: Re: [rddp] STag Invalidation Question To: Stephen Bailey , rddp@ietf.org X-Priority: 3 In-Reply-To: <20030222050923.D83F64EFB@sandmail.sandburst.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/22/03, Stephen Bailey wrote: > >This discussion recurs frequently enough that I think once >the protocol documents have settled and we've got some >miles on the implementations, we (somebody) should >probably write a BCP on how to do safe/defensive ULP >mapping to RDDP, and some guidelines on when to strive for >safety in a ULP mapping/application. > I would suggest the following: The ULP MUST have a policy unambiguously establishing the right to update each portion of an advertised buffer. The ULP MUST NOT be required to disclose that policy. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sat Feb 22 15:36:09 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA15229 for ; Sat, 22 Feb 2003 15:36:09 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1MKhot04569 for rddp-archive@odin.ietf.org; Sat, 22 Feb 2003 15:43:50 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1MKhop04566 for ; Sat, 22 Feb 2003 15:43:50 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA15222 for ; Sat, 22 Feb 2003 15:35:38 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1MKh6p04543; Sat, 22 Feb 2003 15:43:06 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1MKgdp04516 for ; Sat, 22 Feb 2003 15:42:39 -0500 Received: from dmz1.silverbacksystems.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA15211 for ; Sat, 22 Feb 2003 15:34:25 -0500 (EST) Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id 94EC0D16D; Sat, 22 Feb 2003 12:38:05 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id 4FDDC1977C; Sat, 22 Feb 2003 12:38:05 -0800 (PST) Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii" Date: Sat, 22 Feb 2003 12:37:52 -0800 From: "Somesh Gupta" Importance: Normal In-Reply-To: Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 18178-106641F5; Sat, 22 Feb 2003 12:37:40 -0800 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id 0096019277; Sat, 22 Feb 2003 12:37:39 -0800 (PST) Subject: RE: [rddp] MPA TCP analysis draft To: "Caitlin Bestler" , X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit I wonder if we are down a slippery slope on this one where the behavior of the ULP (and its externally visible behavior) starts to depend on whether a TCP packet was dropped or not. Even if this was acceptable from the protocol (TCP based ULP at that) perpectice, all headers that did have completion information in them have to be buffered (or some reduction of them). Also perhaps all ULP rules that may not be enforced under packet drop scenarios vs when everything is in order should be removed from the specification and converted into loose guidelines. Somesh > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Saturday, February 22, 2003 10:03 AM > To: Somesh Gupta; rddp@ietf.org > Subject: RE: [rddp] MPA TCP analysis draft > > > On 2/21/03, Somesh Gupta wrote: > > >1. There is an assumption that the DDP & RDMA headers are > >not needed or buffered elsewhere? For an implementation > >that wants to check for and protect against violation of > >the DDP/RDMA protocols, the headers will be needed later > >when the gaps have been filled in (an example is that > >untagged MSN must be in increasing order). The oft > >repeated rule is out of order placement and not out of > >order delivery. > > I presume you are talking about something like receiving > two DDP segments which are back-to-back in LLP ordering > for the DDP stream, but claim to finish MSN: N and MSN: N+2. > > The receiving DDP MAY detect that MSN N+1 can never arrive > and therefore the stream is in error and should be > terminated. > > I would suggest that it SHOULD have better uses for its > processing capacity. > > Rules requiring MSNs to be in sequence are intended to allow > the receiver to optimize its code, not to add to a list of > validity checks that it MUST perform. > > So if the receiving DDP notes that it cannot properly > generate a completion to the ULP because there is > a gap in the MSNs, it should definitely terminate > the connection and report an error. But that's detecting > the error as a by-product of its normal operations. > Similarly, A duplicate MSN should be detected, because the > first time the final segment of MSN N is received it should > result in completion of the pre-posted receive operation. > The second one would have nothing to complete, hence BOING. > > The receiver is only obligated to validate its peer run-time > compliance with the spec when out-of-spec behaviour could > be used to compromise system security. But Out-of-sequence > MSNs do not place the receiver's resources at risk. It > cannot be used to obtain access to any ULP buffers that > could not have been accessed in order. > > LLP checks against improper sequencing are common because > they are a method of spotting forged packets. But I would > submit that this is an LLP responsibility, DDP should > presume that the packet passed to it is a valid part > of the DDP stream and not attempt to do the LLP's job. > > > > > > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sat Feb 22 16:26:57 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15900 for ; Sat, 22 Feb 2003 16:26:57 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1MLYeu06349 for rddp-archive@odin.ietf.org; Sat, 22 Feb 2003 16:34:40 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1MLYdp06346 for ; Sat, 22 Feb 2003 16:34:39 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15893 for ; Sat, 22 Feb 2003 16:26:26 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1MLY1p06336; Sat, 22 Feb 2003 16:34:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1MLWVp06287 for ; Sat, 22 Feb 2003 16:32:31 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA15824 for ; Sat, 22 Feb 2003 16:24:18 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 91C0C244CC3; Sat, 22 Feb 2003 16:28:08 -0500 (EST) Date: Sat, 22 Feb 2003 15:29:30 -0600 From: Caitlin Bestler Subject: RE: [rddp] MPA TCP analysis draft To: Somesh Gupta Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/22/03, Somesh Gupta wrote: >I wonder if we are down a slippery slope on this one >where the behavior of the ULP (and its externally >visible behavior) starts to depend on whether a TCP >packet was dropped or not. > DDP runs over a reliable IP-based LLP. If an LLP packet has been lost, the connection will be terminated. As with any LLP error, the timing of when that error will be reported to the local ULP is not determined. In that sense the ULP behavior is "dependent" on the handling. But in the sense that the connection will be terminated the behavior is totally consistent. LLP enforcement of LLP rules is not an issue for RDMAP/DDP. Further the receiving RDAMP/DDP is not required to dynamically check that its peer is compliant with the protocol's specifications unless failure to do so exposes the host to an attack. I cannot come up with any scenario where playing games with DDP segments can compromise memory that had not already been exposed by the ULP and was known by the ULP to be exposed to the remote peer. Similarly, how one ULP peer protects itself against its alleged collaborator is a ULP issue, not an RDMAP/DDP one. There are rules that MUST be enforced. For example, after delivering the untagged message X which invalidated STag Y any use of STag Y MUST be rejected. Event if it pretends to have a lower MSN. But it is also mostly irrelevant. Each untagged message will have a unique MSN (within a wrap-around cycle), and it will generate exactly one completion to the ULP. Further, reception of all untagged messages MUST be pre-authorized by the ULP. So there will only be a small range of valid MSNs at any given time, and inherently each one can only have a single final segment. Those are all by-products of maintaining a ReceiveQ (or some equivalent thereof). There is no need to retain headers of DDP Segments after their data has been placed. That applies to untagged segments, not just tagged ones. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sun Feb 23 10:04:51 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA08597 for ; Sun, 23 Feb 2003 10:04:50 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1NFCuM08228 for rddp-archive@odin.ietf.org; Sun, 23 Feb 2003 10:12:56 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1NFCup08225 for ; Sun, 23 Feb 2003 10:12:56 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA08582 for ; Sun, 23 Feb 2003 10:04:19 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1NFCCp08215; Sun, 23 Feb 2003 10:12:12 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1NFBHp08167 for ; Sun, 23 Feb 2003 10:11:17 -0500 Received: from palrel12.hp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA08530 for ; Sun, 23 Feb 2003 10:02:34 -0500 (EST) Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel12.hp.com (Postfix) with ESMTP id 560A11C0197B for ; Sun, 23 Feb 2003 07:06:28 -0800 (PST) Received: from mk731915.cup.hp.com ([15.113.161.2]) by esmail.cup.hp.com (8.9.3/8.8.6) with ESMTP id HAA12288 for ; Sun, 23 Feb 2003 07:03:47 -0800 (PST) Message-Id: <5.1.0.14.2.20030223064726.020367e0@esmail.cup.hp.com> X-Sender: krause@esmail.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Sun, 23 Feb 2003 07:02:09 -0800 To: From: Michael Krause Subject: RE: [rddp] MPA TCP analysis draft In-Reply-To: References: Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 04:20 PM 2/21/2003 -0800, Somesh Gupta wrote: >I wanted to explore the assumption behind the memory sizes somewhat >more. > >The formulae given are > >BufferSizeNAF = K1 * EMSS * number_of_connections + K2 * EMSS > >and > >BufferSizeAF = K2 * EMSS > >There is no question that overall there is a lot of simplification >if there is header alignment, and there is only upper layer PDU >per TCP segment (BTW by having multiple Upper Layer PDUs per >segment, a lot of the simplification has been lost). > >However, it is important that the protocol not be optimized only >around this, since alignment cannot be maintained by software >based senders without modifications to the TCP stack. I pointed >out references to the BSD TCP software in one of my previous emails, >and can repeat it if needed. > >Now to discussing the formuale: > >1. There is an assumption that the DDP & RDMA headers are not >needed or buffered elsewhere? For an implementation that >wants to check for and protect against violation of the DDP/RDMA >protocols, the headers will be needed later when the gaps have >been filled in (an example is that untagged MSN must be in >increasing order). The oft repeated rule is out of order placement >and not out of order delivery. Implementations clearly do not have to buffer any headers - they only need to understand the number of in-flight messages and know when the LLP window has been filled in such that a given message can be delivered. Tracking of such information can be done, for example, within the posted work request which is used by the implementation to describe the associated physical buffer if desired. This is again an implementation option, and as one can see, it is fairly straight forward to implement in hardware - there are many other implementation choices one could use to solve this type of problem without requiring the headers to be maintained. >We can make a number of assumptions here. > >a. It is not necessary as we don't expect this to happen. >b. Our NIC is only a TCP Direct Data Placement NIC and defers > all DDP/RDMA processing to the host One of the objectives of the RDMA / DDP protocols is to allow a source endnode to interact with remote memory without requiring any remote interaction. While an implementation is not precluded from having remote interaction (cannot be seen at the wire protocol level), one does take away some of the customer visible value propositions if the remote processor is used in any way to perform a RDMA Write, RDMA Read, or SEND operation as this reduces the value of the functionality provided by RDMA/DDP as well as the potential system / application performance. >c. We have to buffer the headers. In such a case, the BufferSize > becomes a function of the number of connections regardless of > the method chosen. (see discussion of number of connections below) > >2. There is an assumption of markers being smaller than the EMSS. >Since markers are essential to locating the header (either in >MPA or in IFT), you have to first find a marker, and only then >can you place the data if segments are out of order. As Divid >pointed out, a smaller marker interval puts an unacceptable >burden on a software client (I thing the example was of a >64KB read/write requiring 140+ markers - each of which causes >creation of that many segments, plus creating and inserting >markers and marker vectors etc) - worse than simply using >a NIC doing a large TCP send or TOE offload. The optimal design is a hardware implementation that provides marker insertion / removal. The objective is to insure that one can implement and interoperate with a software-based solution without creating a complete performance nightmare but also with the understanding that a software-based solution will not perform as well as a hardware-based solution. The current draft specifications attempt to strike a good balance in allowing high-performance hardware implementations while insuring software-based can be constructed. I know of at least a couple of software implementations that are in-flight today and no one is stating the use of markers or its proposed interval are a problem. >a. Either severely penalize software based senders (at hopefully >your own risk rather than institutionalizing the risk via the >protocol). > >b. or alternatively add a factor of > K1 * Marker Interval/EMSS * number_of_connections to the >buffering needs. > >3. The number of connections in the above equations is really >the number of connections with out of order segments - and not >the total number of connections. Of course, for this you would >have to have the separate header and data CRC as proposed by Jim. >Hopefully this is a much smaller number than the number of >connections. > >Of course, if every other segment is out of order, one does not have >worry about buffering or memory speeds etc because TCP algorithms >will force you to a rate in the range of KBytes and not Gbits. >TCP is not a credit based mechanism but uses the AIMD algorithms to >"detect" network congestion. It will assume that it needs to back >off significantly to prevent a congestion collapse of the network. > >Somesh > > > > -----Original Message----- > > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > > Caitlin Bestler > > Sent: Tuesday, February 18, 2003 1:55 PM > > To: rddp@ietf.org > > Subject: Re: [rddp] MPA TCP analysis draft > > > > > > The MPA analysis draft offers the following formula starting > > on page 10: > > > > > For the case when the FPDU is not aligned with the TCP > > > segment, it may take, on average, 2 TCP segments to > > > assemble one FPDU. Therefore, the receiver has to > > > allocate BufferSizeNAF (Buffer Size, Non-Aligned FPDU) > > > octets: > > > > > > BufferSizeNAF = K1* EMSS * number_of_connections + > > > K2 * EMSS > > > > > > > > > Where K1 and K2 are implementation dependent constants > > > and EMSS is the effective maximum segment size. > > > > > > For example, a 1 Gbps link with 10,000 connections and an > > > EMSS of 1500B would require 15 MB of memory. Often the > > > number of connections used scales with the network speed, > > > aggravating the situation for higher speeds. > > > > > > The derivation of this formula does not seem to be supported > > anywhere in the text. > > > > In fact I can't figure out where it came from, or what > > assumptions are being swept under the rug in K1 and K2. > > > > As pointed out, 2 adjacent TCP segments should allow the > > assembly of one FPDU. (In fact it is only the strangest > > of corner cases where this is not true). So the question > > becomes, how many TCP segments might have to be buffered > > before an FPDU is assembled and can be placed? > > > > Phrased that way, the answer is obviously half of the > > TCP segments that can be in flight. The worst case scenario > > is clearly dropping every other TCP segment. (And as > > previously noted, if a load-balancing router dispatches > > successive segments by alternate path that is *exactly* what > > could happen if one path is delayed). > > > > I believe that is a simpler explanation, and formula. > > It does not change any of the conclusions, however. > > > > If the formula presented actually represent a *lower* > > estimate of the required buffering then the rationale > > should be explained. > > > > Caitlin Bestler > > http://asomi.com/CaitlinBestler/ > > _______________________________________________ > > rddp mailing list > > rddp@ietf.org > > https://www1.ietf.org/mailman/listinfo/rddp >_______________________________________________ >rddp mailing list >rddp@ietf.org >https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sun Feb 23 11:06:02 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA09247 for ; Sun, 23 Feb 2003 11:06:01 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1NGE7311073 for rddp-archive@odin.ietf.org; Sun, 23 Feb 2003 11:14:07 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1NGE7p11070 for ; Sun, 23 Feb 2003 11:14:07 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA09236 for ; Sun, 23 Feb 2003 11:05:30 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1NGDBp11051; Sun, 23 Feb 2003 11:13:11 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1NGCEp11036 for ; Sun, 23 Feb 2003 11:12:14 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA09230 for ; Sun, 23 Feb 2003 11:03:38 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id IAA25966 for ; Sun, 23 Feb 2003 08:07:29 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id <105KW5M8>; Sun, 23 Feb 2003 11:07:29 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FF7@xbl.ma.emulex.com> From: "Williams, Jim" To: rddp@ietf.org Date: Sun, 23 Feb 2003 11:07:28 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] I-D ACTION:draft-williams-iwarp-ift-01.txt Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This draft is pretty rough, but contains a few new ideas. It is certainly not a finished proposal, but I hope it may stimulate some discussion. > -----Original Message----- > From: Internet-Drafts@ietf.org [mailto:Internet-Drafts@ietf.org] > Sent: Wednesday, February 19, 2003 10:05 AM > Subject: I-D ACTION:draft-williams-iwarp-ift-01.txt > > > A New Internet-Draft is available from the on-line Internet-Drafts > directories. > > A URL for this Internet-Draft is: > http://www.ietf.org/internet-drafts/draft-williams-iwarp-ift-01.txt > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sun Feb 23 11:35:34 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA09546 for ; Sun, 23 Feb 2003 11:35:33 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1NGhdq12461 for rddp-archive@odin.ietf.org; Sun, 23 Feb 2003 11:43:39 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1NGhdp12458 for ; Sun, 23 Feb 2003 11:43:39 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA09543 for ; Sun, 23 Feb 2003 11:35:02 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1NGh1p12440; Sun, 23 Feb 2003 11:43:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1NGgYp12411 for ; Sun, 23 Feb 2003 11:42:34 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA09512 for ; Sun, 23 Feb 2003 11:33:57 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id IAA26244 for ; Sun, 23 Feb 2003 08:37:49 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id <105KW5NQ>; Sun, 23 Feb 2003 11:37:48 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FF8@xbl.ma.emulex.com> From: "Williams, Jim" To: rddp@ietf.org Subject: RE: [rddp] IFT-01 - SP (Speculative Placement) Flag Date: Sun, 23 Feb 2003 11:37:47 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Thursday, February 20, 2003 11:16 PM > To: rddp@ietf.org > Subject: [rddp] IFT-01 - SP (Speculative Placement) Flag > > > The proposed flags include: > > > SP Indicates the sender is giving the receiver > > permission to do speculative placement. If this bit is > > set, the receiver MAY do speculative placement. If not > > set, the receiver MUST NOT do speculative placement. See > > the chapter below for full description of speculative > > placement. > > Why would the *sender* have an interest in mandating that > the receiver not take risks with the *receiver's* memory? I am sure the draft could have been better at explaining this. The important point is that the receiver is taking *no risk* with its memory as long as the ULP follows the rules for speculative placement. These rules are basically: 1. De-register (remove remote write access) to all buffers before allowing the ULP to look at them. 2. Do not write a memory location more that once during the period that a buffer is remote write enabled. This may be necessary anyway if out of order placement is to have deterministic results. 3. Do not assume anything about the contents of unwritten memory locations within buffers that have been registered for remote write access. So the ULP knows that as long as these rules are followed, speculative placement works and is risk free. The flag bit is arguably redundant since the ULP already knows if it supports speculative placement, however like much of the proposed adaption indication, it is a redundant confirmation designed to reduce risk of a mix up between the two ends. > Can the sender specify that the receiver not take risks with > the data by running certain operating systems? Huh? Which operating systems do you think are too riksy? :) _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Sun Feb 23 17:44:04 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14196 for ; Sun, 23 Feb 2003 17:44:03 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1NMqHi29598 for rddp-archive@odin.ietf.org; Sun, 23 Feb 2003 17:52:17 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1NMqHp29593 for ; Sun, 23 Feb 2003 17:52:17 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14175 for ; Sun, 23 Feb 2003 17:43:32 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1NMpLp29557; Sun, 23 Feb 2003 17:51:21 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1NMojp29526 for ; Sun, 23 Feb 2003 17:50:45 -0500 Received: from palrel12.hp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14130 for ; Sun, 23 Feb 2003 17:42:01 -0500 (EST) Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel12.hp.com (Postfix) with ESMTP id 475E91C01171 for ; Sun, 23 Feb 2003 14:45:53 -0800 (PST) Received: from mk731915.cup.hp.com ([15.113.161.2]) by esmail.cup.hp.com (8.9.3/8.8.6) with ESMTP id OAA04681 for ; Sun, 23 Feb 2003 14:42:45 -0800 (PST) Message-Id: <5.1.0.14.2.20030223143722.03b6dc20@esmail.cup.hp.com> X-Sender: krause@esmail.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Sun, 23 Feb 2003 14:45:19 -0800 To: rddp@ietf.org From: Michael Krause Subject: RE: [rddp] IFT-01 - SP (Speculative Placement) Flag In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B8FF8@xbl.ma.emulex.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 11:37 AM 2/23/2003 -0500, Williams, Jim wrote: > > -----Original Message----- > > From: Caitlin Bestler [mailto:cait@asomi.com] > > Sent: Thursday, February 20, 2003 11:16 PM > > To: rddp@ietf.org > > Subject: [rddp] IFT-01 - SP (Speculative Placement) Flag > > > > > > The proposed flags include: > > > > > SP Indicates the sender is giving the receiver > > > permission to do speculative placement. If this bit is > > > set, the receiver MAY do speculative placement. If not > > > set, the receiver MUST NOT do speculative placement. See > > > the chapter below for full description of speculative > > > placement. > > > > Why would the *sender* have an interest in mandating that > > the receiver not take risks with the *receiver's* memory? > > >I am sure the draft could have been better at explaining this. >The important point is that the receiver is taking *no risk* >with its memory as long as the ULP follows the rules for >speculative placement. These rules are basically: > > 1. De-register (remove remote write access) to all buffers > before allowing the ULP to look at them. Why? The ULP owns the buffers and once delivered, it knows the contents correct from the LLP / DDP / RDMA perspective. If someone wants to invalidate access rights, then it is a ULP issue and not a DDP/RDMA issue to resolve. DDP/RDMA provides an invalidate option to close this update capability if desired. > 2. Do not write a memory location more that once during the > period that a buffer is remote write enabled. This may > be necessary anyway if out of order placement is to > have deterministic results. Once a buffer is advertised, it is up to the LLP to insure that duplicates are not placed thus there is no value in providing such a capability as this is truly redundant to the LLP responsibilities. If one is concerned about a partial overwrite by a subsequent operation, then a sender-based FENCE should be used to insure that the operation is completed (can be done as a RDMA Read or a delay in processing the next operation until the window progresses to indicate the previous operation has been placed). In general, buffer state in non-deterministic until the window has progressed to indicate everything has been delivered. > 3. Do not assume anything about the contents of unwritten > memory locations within buffers that have been > registered for remote write access. If the memory has not been written to and the contents were not the target of any update (i.e. the consumer has not advertised the memory for update by any operation), then the memory is clearly known even if it is registered. Registration just states the virtual-to-physical or the physical addresses are capable of being accessed - it does not change the state of the buffers or their ability to be coherently accessed by an application or a ULP. >So the ULP knows that as long as these rules are followed, >speculative placement works and is risk free. The flag bit >is arguably redundant since the ULP already knows if it >supports speculative placement, however like much of the >proposed adaption indication, it is a redundant confirmation >designed to reduce risk of a mix up between the two ends. It does not add value from what you have described here. Mike _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 09:44:13 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA15497 for ; Mon, 24 Feb 2003 09:44:13 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OEqle29741 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 09:52:47 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OEqlp29738 for ; Mon, 24 Feb 2003 09:52:47 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA15481 for ; Mon, 24 Feb 2003 09:43:42 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OEq7p29680; Mon, 24 Feb 2003 09:52:07 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OEbNp29055 for ; Mon, 24 Feb 2003 09:37:23 -0500 Received: from mxic2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA15238 for ; Mon, 24 Feb 2003 09:28:18 -0500 (EST) From: Black_David@emc.com Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRHVBHH>; Mon, 24 Feb 2003 09:32:12 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C919@corpmx14.us.dg.com> To: rddp@ietf.org Date: Mon, 24 Feb 2003 09:29:39 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] Marker Interval Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Somesh Gupta wrote: > >As David pointed out, a smaller marker > >interval puts an unacceptable burden on a software client > >(I thing the example was of a 64KB read/write requiring > >140+ markers - each of which causes creation of that many > >segments, plus creating and inserting markers and marker > >vectors etc) - worse than simply using a NIC doing a large > >TCP send or TOE offload. That's definitely a problem. Caitlin Bestler wrote: > The pattern where the true payload is a repeating pattern. > 140 markers does not require 140 distinct descriptors. > If you look up old mini-computers you'll find many examples > of repeating groups in DMA control. That's mostly a statement about building new hardware. A lot of existing DMA hardware, OS syscall interfaces, etc. don't support repeating descriptors. Michael Krause wrote: > The optimal design is a hardware implementation that provides marker > insertion / removal. The objective is to insure that one can implement > and interoperate with a software-based solution without creating a > complete performance nightmare but also with the understanding that a > software-based solution will not perform as well as a hardware-based > solution. The 512-byte fixed marker interval appears to have been over-optimized for optimal (new hardware) designs to the detriment of (re)use of existing software stacks. The design rationale that I've seen based on minimum required IP payload sizes and the desire for a "flow-through" implementation is not very convincing, especially as the current MPA design can't achieve "flow-through" due to the inability for a receiver to determine whether an aligned stream will stay aligned (yes, one could build a "flow-through" implementation, but it will have robustness problems akin to a TCP implementation with inadequate buffering). In addition, we can go back through the "there's no such thing as a buffer- less [R]NIC discussion" if that's necessary, but more to the point is that the number of descriptors for reasonable-sized operations (e.g., 16 or more for a single 8KB transfer) is hard to justify. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 10:05:05 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA16095 for ; Mon, 24 Feb 2003 10:05:05 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OFDeM31597 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 10:13:40 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OFDep31594 for ; Mon, 24 Feb 2003 10:13:40 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA16092 for ; Mon, 24 Feb 2003 10:04:34 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OFD1p31559; Mon, 24 Feb 2003 10:13:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OFC2p31520 for ; Mon, 24 Feb 2003 10:12:02 -0500 Received: from palrel10.hp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA16075 for ; Mon, 24 Feb 2003 10:02:56 -0500 (EST) Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel10.hp.com (Postfix) with ESMTP id 27ECC1C0164B for ; Mon, 24 Feb 2003 07:06:50 -0800 (PST) Received: from mk731915.cup.hp.com ([15.113.161.2]) by esmail.cup.hp.com (8.9.3/8.8.6) with ESMTP id HAA24343 for ; Mon, 24 Feb 2003 07:04:09 -0800 (PST) Message-Id: <5.1.0.14.2.20030224065953.0200bb10@esmail.cup.hp.com> X-Sender: krause@esmail.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Mon, 24 Feb 2003 07:06:44 -0800 To: rddp@ietf.org From: Michael Krause Subject: Re: [rddp] Marker Interval In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C919@corpmx14.us.dg.co m> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 09:29 AM 2/24/2003 -0500, Black_David@emc.com wrote: >Somesh Gupta wrote: > > > >As David pointed out, a smaller marker > > >interval puts an unacceptable burden on a software client > > >(I thing the example was of a 64KB read/write requiring > > >140+ markers - each of which causes creation of that many > > >segments, plus creating and inserting markers and marker > > >vectors etc) - worse than simply using a NIC doing a large > > >TCP send or TOE offload. > >That's definitely a problem. > >Caitlin Bestler wrote: > > > The pattern where the true payload is a repeating pattern. > > 140 markers does not require 140 distinct descriptors. > > If you look up old mini-computers you'll find many examples > > of repeating groups in DMA control. > >That's mostly a statement about building new hardware. A lot of >existing DMA hardware, OS syscall interfaces, etc. don't support >repeating descriptors. > >Michael Krause wrote: > > > The optimal design is a hardware implementation that provides marker > > insertion / removal. The objective is to insure that one can implement > > and interoperate with a software-based solution without creating a > > complete performance nightmare but also with the understanding that a > > software-based solution will not perform as well as a hardware-based > > solution. > >The 512-byte fixed marker interval appears to have been over-optimized >for optimal (new hardware) designs to the detriment of (re)use of >existing software stacks. The design rationale that I've seen based >on minimum required IP payload sizes and the desire for a "flow-through" >implementation is not very convincing, especially as the current MPA >design can't achieve "flow-through" due to the inability for a receiver >to determine whether an aligned stream will stay aligned (yes, one >could build a "flow-through" implementation, but it will have robustness >problems akin to a TCP implementation with inadequate buffering). In >addition, we can go back through the "there's no such thing as a buffer- >less [R]NIC discussion" if that's necessary, but more to the point is >that the number of descriptors for reasonable-sized operations >(e.g., 16 or more for a single 8KB transfer) is hard to justify. The 512-byte fixed marker interval proposed was based on requirement to be compliant with existing RFCs, analysis of packet sizes relative to targeted workloads, ability to efficiently implement in hardware, etc. Depending upon the software stack implementation, the insertion of markers can be accomplished at the same time one is copying / calculating the checksum for the packet thus it is relatively isolated within the stack. This was verified by examination of various OSV offerings including active support by multiple OSVs who stated this was not an issue after they had examined their implementations to validate the proposal and its cost of implementation. The consensus of many involved in the draft development was that due diligence was done in examining the impacts to software and make the right trade-off between the two types of implementations. Mike _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 10:44:14 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA17174 for ; Mon, 24 Feb 2003 10:44:14 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OFqoC01610 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 10:52:50 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OFqnp01607 for ; Mon, 24 Feb 2003 10:52:49 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA17167 for ; Mon, 24 Feb 2003 10:43:43 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OFq8p01582; Mon, 24 Feb 2003 10:52:08 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OFphp01546 for ; Mon, 24 Feb 2003 10:51:43 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA17155 for ; Mon, 24 Feb 2003 10:42:37 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id HAA10859 for ; Mon, 24 Feb 2003 07:46:30 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id <105KW6VP>; Mon, 24 Feb 2003 10:46:29 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B8FFC@xbl.ma.emulex.com> From: "Williams, Jim" To: rddp@ietf.org Date: Mon, 24 Feb 2003 10:46:29 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] IFT-01 - SP (Speculative Placement) Flag Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > It does not add value from what you have described here. > > Mike Let me try to frame this discussion a bit. The goal of speculative placement is to achieve the following. Allow the receiver to do direct placement without markers or guaranteed alignment. This is done not by absolutely insuring that the receiver can identify DDP headers, but rather by allowing the receiver to get it right almost all the time, but providing a mechanism to "fix up" any damage done by getting it wrong on rare occasions. The "fix up" would be transparent to the ULP. The objective of the above is to allow the benefits of direct placement even when the sender is an unmodified TCP stack, and without requiring such a stack to incur the overhead of inserting markers. Clean and performant interoperability with existing TCP implementations is an area where the MPA proposal is clearly deficient. SP is an effort to explore solutions to this. Having said that, I believe the proposal for speculative placement outlined in the draft comes up short in some ways. So I don't want to defend it as _THE_ solution the problem. I did feel it was worth putting on the table for discussion, though. But this discussion may be uninteresting to those the feel performant interoperability with existing TCP is not a goal. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 10:54:08 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA17446 for ; Mon, 24 Feb 2003 10:54:08 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OG2ig02131 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 11:02:44 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OG2ip02128 for ; Mon, 24 Feb 2003 11:02:44 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA17427 for ; Mon, 24 Feb 2003 10:53:37 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OG22p02087; Mon, 24 Feb 2003 11:02:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OG1hp02057 for ; Mon, 24 Feb 2003 11:01:43 -0500 Received: from ztxmail05.ztx.compaq.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA17394 for ; Mon, 24 Feb 2003 10:52:35 -0500 (EST) Received: from cceexg12.americas.cpqcorp.net (cceexg12.americas.cpqcorp.net [16.110.250.124]) by ztxmail05.ztx.compaq.com (Postfix) with ESMTP id A867F3A78; Mon, 24 Feb 2003 09:56:17 -0600 (CST) Received: from cceexc16.americas.cpqcorp.net ([16.110.250.58]) by cceexg12.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Mon, 24 Feb 2003 09:56:11 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: RE: [rddp] Marker Interval Date: Mon, 24 Feb 2003 09:56:11 -0600 Message-ID: <01CA1B1F74A4EC40A9C887F7B58F8724031430DE@cceexc16.americas.cpqcorp.net> Thread-Topic: [rddp] Marker Interval Thread-Index: AcLcFMsuqJLB6NWSRfWckEwROaZE+wABpldA From: "Barron, Dwight" To: , X-OriginalArrivalTime: 24 Feb 2003 15:56:11.0697 (UTC) FILETIME=[40ABFA10:01C2DC1D] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h1OG1hp02058 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit Assuming we had a large number of negotiable marker intervals, what would the optimum negotiation strategy be for a software-MPA to connect to a hardware-MPA implementation? Clearly, the SW-MPA would want larger intervals to minimize overhead, while presumably the HW-MPA would want smaller intervals to minimize buffering. While I see a lot of value in having mixed mode connections, I keep coming back the notion that the SW implementation is willingly taking on additional overhead for the benefit of the HW implementation, in which case the SW end would accept the HW's recommended marker interval value. Regards, Dwight > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Monday, February 24, 2003 8:30 AM > To: rddp@ietf.org > Subject: [rddp] Marker Interval > > > Somesh Gupta wrote: > > > >As David pointed out, a smaller marker > > >interval puts an unacceptable burden on a software client > > >(I thing the example was of a 64KB read/write requiring > > >140+ markers - each of which causes creation of that many > > >segments, plus creating and inserting markers and marker > > >vectors etc) - worse than simply using a NIC doing a large > > >TCP send or TOE offload. > > That's definitely a problem. > > Caitlin Bestler wrote: > > > The pattern where the true payload is a repeating pattern. > > 140 markers does not require 140 distinct descriptors. > > If you look up old mini-computers you'll find many examples > > of repeating groups in DMA control. > > That's mostly a statement about building new hardware. A lot of > existing DMA hardware, OS syscall interfaces, etc. don't support > repeating descriptors. > > Michael Krause wrote: > > > The optimal design is a hardware implementation that > provides marker > > insertion / removal. The objective is to insure that one > can implement > > and interoperate with a software-based solution without creating a > > complete performance nightmare but also with the > understanding that a > > software-based solution will not perform as well as a hardware-based > > solution. > > The 512-byte fixed marker interval appears to have been over-optimized > for optimal (new hardware) designs to the detriment of (re)use of > existing software stacks. The design rationale that I've seen based > on minimum required IP payload sizes and the desire for a > "flow-through" > implementation is not very convincing, especially as the current MPA > design can't achieve "flow-through" due to the inability for > a receiver > to determine whether an aligned stream will stay aligned (yes, one > could build a "flow-through" implementation, but it will have > robustness > problems akin to a TCP implementation with inadequate buffering). In > addition, we can go back through the "there's no such thing > as a buffer- > less [R]NIC discussion" if that's necessary, but more to the point is > that the number of descriptors for reasonable-sized operations > (e.g., 16 or more for a single 8KB transfer) is hard to justify. > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > black_david@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 11:13:07 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA17845 for ; Mon, 24 Feb 2003 11:13:07 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OGLhf03679 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 11:21:43 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OGLhp03676 for ; Mon, 24 Feb 2003 11:21:43 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA17838 for ; Mon, 24 Feb 2003 11:12:35 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OGL1p03639; Mon, 24 Feb 2003 11:21:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OGKfp03621 for ; Mon, 24 Feb 2003 11:20:41 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA17807 for ; Mon, 24 Feb 2003 11:11:34 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 770C02452EA; Mon, 24 Feb 2003 11:15:27 -0500 (EST) Date: Mon, 24 Feb 2003 10:16:53 -0600 From: Caitlin Bestler Subject: Re: [rddp] IFT-01 - SP (Speculative Placement) Flag To: "Williams, Jim" Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B8FFC@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/24/03, Williams, Jim wrote: > >> It does not add value from what you have described here. >> >> Mike > >Let me try to frame this discussion a bit. > >The goal of speculative placement is to achieve the >following. > > Allow the receiver to do direct placement without markers > or guaranteed alignment. This is done not by absolutely > insuring that the receiver can identify DDP headers, but > rather by allowing the receiver to get it right almost > all the time, but providing a mechanism to "fix up" any > damage done by getting it wrong on rare occasions. The > "fix up" would be transparent to the ULP. > > The objective of the above is to allow the benefits of > direct placement even when the sender is an unmodified > TCP stack, and without requiring such a stack to incur > the overhead of inserting markers. > > Clean and performant interoperability with existing TCP > implementations is an area where the MPA proposal is > clearly deficient. SP is an effort to explore > solutions to this. > >Having said that, I believe the proposal for speculative >placement outlined in the draft comes up short in some ways. >So I don't want to defend it as _THE_ solution the problem. >I did feel it was worth putting on the table for discussion, >though. > >But this discussion may be uninteresting to those the >feel performant interoperability with existing TCP is >not a goal. Let me take a stab at this. Speculative Placement is a method that allows DDP Segments to be found within out-of-order TCP segments without requiring the use of markers. Many practices can make the probability of finding the DDP Segment quickly very high: aligning at the transmit side, and long stretches where the DDP segments are all the maximum size. However, none of these methods are perfect. Therefore if this technique is to be used, it is vital that any previously delivered payload be protected from accidental overwrite. Speculative placement should only be used when ALL exposed buffers have had NO deliveries since they were enabled. (i.e., the ULP is invalidating the STag with *each* untagged message). Is my analysis of the applicability/intent correct? Any form of variable marker interval can lead to situations where speculative placement is a valuable MAY to be included in the draft. There is still a substantial question in my mind as to whether the value of variable marker interval justifies their complexity. I believe everyone agrees that markers are ugly. But there seems to be varying opinion on whether they are an impediment to software implemntations and/or implemntations that attempt to optimize conventional TCP streams at the same time. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 12:17:14 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA19482 for ; Mon, 24 Feb 2003 12:17:13 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OHPnn08117 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 12:25:49 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OHPnp08114 for ; Mon, 24 Feb 2003 12:25:49 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA19476 for ; Mon, 24 Feb 2003 12:16:42 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OHP5p08093; Mon, 24 Feb 2003 12:25:05 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OHO1p08018 for ; Mon, 24 Feb 2003 12:24:01 -0500 Received: from mxic1.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA19445 for ; Mon, 24 Feb 2003 12:14:54 -0500 (EST) From: Black_David@emc.com Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1KDZDFG8>; Mon, 24 Feb 2003 12:17:55 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C923@corpmx14.us.dg.com> To: krause@cup.hp.com, rddp@ietf.org Date: Mon, 24 Feb 2003 12:16:03 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] Markers and Software Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > Depending upon the software stack implementation, the insertion of markers > can be accomplished at the same time one is copying / calculating the > checksum for the packet thus it is relatively isolated within the > stack. This was verified by examination of various OSV offerings > including active support by multiple OSVs who stated this was not an issue > after they had examined their implementations to validate the proposal and > its cost of implementation. That is software, but not an unmodified existing stack, which is of importance here, in contrast to the environment in which the MPA draft was originally developed. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 12:22:11 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA19600 for ; Mon, 24 Feb 2003 12:22:11 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OHUln08371 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 12:30:47 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OHUkp08368 for ; Mon, 24 Feb 2003 12:30:46 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA19578 for ; Mon, 24 Feb 2003 12:21:39 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OHU2p08313; Mon, 24 Feb 2003 12:30:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OHTep08257 for ; Mon, 24 Feb 2003 12:29:40 -0500 Received: from dmz1.silverbacksystems.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA19551 for ; Mon, 24 Feb 2003 12:20:32 -0500 (EST) Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id AEF10D145; Mon, 24 Feb 2003 09:24:12 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id 4879019123; Mon, 24 Feb 2003 09:24:12 -0800 (PST) Cc: Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii" Date: Mon, 24 Feb 2003 09:24:11 -0800 From: "Somesh Gupta" Importance: Normal In-Reply-To: Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 29119-38954EA8; Mon, 24 Feb 2003 09:23:58 -0800 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id F401E16E85; Mon, 24 Feb 2003 09:23:57 -0800 (PST) Subject: RE: [rddp] MPA TCP analysis draft To: "Caitlin Bestler" X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit > -----Original Message----- > From: Caitlin Bestler [mailto:cait@asomi.com] > Sent: Saturday, February 22, 2003 1:30 PM > To: Somesh Gupta > Cc: rddp@ietf.org > Subject: RE: [rddp] MPA TCP analysis draft > > > On 2/22/03, Somesh Gupta wrote: > > >I wonder if we are down a slippery slope on this one > >where the behavior of the ULP (and its externally > >visible behavior) starts to depend on whether a TCP > >packet was dropped or not. > > > DDP runs over a reliable IP-based LLP. If an LLP packet > has been lost, the connection will be terminated. Dropped not lost. Dropped (and subsequently recovered) packets should not be visible to the ULP, much less have an impact on its behavior. > > As with any LLP error, the timing of when that error will > be reported to the local ULP is not determined. In that > sense the ULP behavior is "dependent" on the handling. > But in the sense that the connection will be terminated > the behavior is totally consistent. I am somewhat confused over here. For TCP dropped and subsequently recovered packets are normal and should not have any impact on ULP behavior. Obviously I am missing a point here. > > LLP enforcement of LLP rules is not an issue for RDMAP/DDP. I was talking about enforcing DDP rules here. > > Further the receiving RDAMP/DDP is not required to > dynamically check that its peer is compliant with the > protocol's specifications unless failure to do so exposes > the host to an attack. That means that it is not important to have those rules in the protocol specification. > > I cannot come up with any scenario where playing games > with DDP segments can compromise memory that had not already > been exposed by the ULP and was known by the ULP to be > exposed to the remote peer. > > Similarly, how one ULP peer protects itself against its > alleged collaborator is a ULP issue, not an RDMAP/DDP one. > > There are rules that MUST be enforced. For example, after > delivering the untagged message X which invalidated STag Y > any use of STag Y MUST be rejected. Event if it pretends to > have a lower MSN. > > But it is also mostly irrelevant. Each untagged message will > have a unique MSN (within a wrap-around cycle), and it will > generate exactly one completion to the ULP. Further, > reception of all untagged messages MUST be pre-authorized by > the ULP. So there will only be a small range of valid MSNs > at any given time, and inherently each one can only have a > single final segment. > > Those are all by-products of maintaining a ReceiveQ (or some > equivalent thereof). There is no need to retain headers of > DDP Segments after their data has been placed. That applies > to untagged segments, not just tagged ones. > > Caitlin Bestler > http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 12:22:14 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA19614 for ; Mon, 24 Feb 2003 12:22:14 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OHUoH08390 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 12:30:50 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OHUop08387 for ; Mon, 24 Feb 2003 12:30:50 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA19582 for ; Mon, 24 Feb 2003 12:21:42 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OHU2p08329; Mon, 24 Feb 2003 12:30:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OHTfp08261 for ; Mon, 24 Feb 2003 12:29:41 -0500 Received: from dmz1.silverbacksystems.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA19552 for ; Mon, 24 Feb 2003 12:20:32 -0500 (EST) Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id B9B2AD14E; Mon, 24 Feb 2003 09:24:12 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id 4965C192D0; Mon, 24 Feb 2003 09:24:12 -0800 (PST) Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii" Date: Mon, 24 Feb 2003 09:24:10 -0800 From: "Somesh Gupta" Importance: Normal In-Reply-To: <5.1.0.14.2.20030223064726.020367e0@esmail.cup.hp.com> Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 29117-5AA6B375; Mon, 24 Feb 2003 09:23:57 -0800 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id B13B716E85; Mon, 24 Feb 2003 09:23:57 -0800 (PST) Subject: RE: [rddp] MPA TCP analysis draft To: "Michael Krause" , X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit > -----Original Message----- > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > Michael Krause > Sent: Sunday, February 23, 2003 7:02 AM > To: rddp@ietf.org > Subject: RE: [rddp] MPA TCP analysis draft > > > At 04:20 PM 2/21/2003 -0800, Somesh Gupta wrote: > >I wanted to explore the assumption behind the memory sizes somewhat > >more. > > > >The formulae given are > > > >BufferSizeNAF = K1 * EMSS * number_of_connections + K2 * EMSS > > > >and > > > >BufferSizeAF = K2 * EMSS > > > >There is no question that overall there is a lot of simplification > >if there is header alignment, and there is only upper layer PDU > >per TCP segment (BTW by having multiple Upper Layer PDUs per > >segment, a lot of the simplification has been lost). > > > >However, it is important that the protocol not be optimized only > >around this, since alignment cannot be maintained by software > >based senders without modifications to the TCP stack. I pointed > >out references to the BSD TCP software in one of my previous emails, > >and can repeat it if needed. > > > >Now to discussing the formuale: > > > >1. There is an assumption that the DDP & RDMA headers are not > >needed or buffered elsewhere? For an implementation that > >wants to check for and protect against violation of the DDP/RDMA > >protocols, the headers will be needed later when the gaps have > >been filled in (an example is that untagged MSN must be in > >increasing order). The oft repeated rule is out of order placement > >and not out of order delivery. > > Implementations clearly do not have to buffer any headers - they > only need > to understand the number of in-flight messages and know when the > LLP window > has been filled in such that a given message can be delivered. > Tracking of > such information can be done, for example, within the posted work request > which is used by the implementation to describe the associated physical > buffer if desired. This is again an implementation option, and > as one can > see, it is fairly straight forward to implement in hardware - there are > many other implementation choices one could use to solve this type of > problem without requiring the headers to be maintained. Clearly some form of information has to be kept (does not have to be headers). If the DDP rules are to be verified, then more information has to be kept. Even if rules are not verified in out of order segments case, the headers or variation thereof that generate completion must be kept. The DDP specification should be changed if rules are not to be verified - otherwise the information to be maintained is of the order of number of connections. Or scoreboarding can be done, which is not likely to be easier. > > >We can make a number of assumptions here. > > > >a. It is not necessary as we don't expect this to happen. > >b. Our NIC is only a TCP Direct Data Placement NIC and defers > > all DDP/RDMA processing to the host > > One of the objectives of the RDMA / DDP protocols is to allow a source > endnode to interact with remote memory without requiring any remote > interaction. While an implementation is not precluded from having remote > interaction (cannot be seen at the wire protocol level), one does > take away > some of the customer visible value propositions if the remote > processor is > used in any way to perform a RDMA Write, RDMA Read, or SEND operation as > this reduces the value of the functionality provided by RDMA/DDP > as well as > the potential system / application performance. > > >c. We have to buffer the headers. In such a case, the BufferSize > > becomes a function of the number of connections regardless of > > the method chosen. (see discussion of number of connections below) > > > >2. There is an assumption of markers being smaller than the EMSS. > >Since markers are essential to locating the header (either in > >MPA or in IFT), you have to first find a marker, and only then > >can you place the data if segments are out of order. As Divid > >pointed out, a smaller marker interval puts an unacceptable > >burden on a software client (I thing the example was of a > >64KB read/write requiring 140+ markers - each of which causes > >creation of that many segments, plus creating and inserting > >markers and marker vectors etc) - worse than simply using > >a NIC doing a large TCP send or TOE offload. > > The optimal design is a hardware implementation that provides marker > insertion / removal. The objective is to insure that one can > implement and > interoperate with a software-based solution without creating a complete > performance nightmare but also with the understanding that a > software-based > solution will not perform as well as a hardware-based solution. The > current draft specifications attempt to strike a good balance in allowing > high-performance hardware implementations while insuring > software-based can > be constructed. I know of at least a couple of software implementations > that are in-flight today and no one is stating the use of markers or its > proposed interval are a problem. I have to disagree with both points over here. Markers do pose a burden on software implementations (vectors have to be split, memory for markers has to be allocated and vectors created for it, number of calls to system have to increase, a number of completions from NIC will increase. The penalty is quite significant. Similarly, the current draft does not allow optimization of RNICs that work well with software based senders (or penalizes it to the point that it is no longer a very viable choice). The IFT draft strikes a much better balance for both environments. > > >a. Either severely penalize software based senders (at hopefully > >your own risk rather than institutionalizing the risk via the > >protocol). > > > >b. or alternatively add a factor of > > K1 * Marker Interval/EMSS * number_of_connections to the > >buffering needs. > > > >3. The number of connections in the above equations is really > >the number of connections with out of order segments - and not > >the total number of connections. Of course, for this you would > >have to have the separate header and data CRC as proposed by Jim. > >Hopefully this is a much smaller number than the number of > >connections. > > > >Of course, if every other segment is out of order, one does not have > >worry about buffering or memory speeds etc because TCP algorithms > >will force you to a rate in the range of KBytes and not Gbits. > >TCP is not a credit based mechanism but uses the AIMD algorithms to > >"detect" network congestion. It will assume that it needs to back > >off significantly to prevent a congestion collapse of the network. > > > >Somesh > > > > > > > -----Original Message----- > > > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > > > Caitlin Bestler > > > Sent: Tuesday, February 18, 2003 1:55 PM > > > To: rddp@ietf.org > > > Subject: Re: [rddp] MPA TCP analysis draft > > > > > > > > > The MPA analysis draft offers the following formula starting > > > on page 10: > > > > > > > For the case when the FPDU is not aligned with the TCP > > > > segment, it may take, on average, 2 TCP segments to > > > > assemble one FPDU. Therefore, the receiver has to > > > > allocate BufferSizeNAF (Buffer Size, Non-Aligned FPDU) > > > > octets: > > > > > > > > BufferSizeNAF = K1* EMSS * number_of_connections + > > > > K2 * EMSS > > > > > > > > > > > > Where K1 and K2 are implementation dependent constants > > > > and EMSS is the effective maximum segment size. > > > > > > > > For example, a 1 Gbps link with 10,000 connections and an > > > > EMSS of 1500B would require 15 MB of memory. Often the > > > > number of connections used scales with the network speed, > > > > aggravating the situation for higher speeds. > > > > > > > > > The derivation of this formula does not seem to be supported > > > anywhere in the text. > > > > > > In fact I can't figure out where it came from, or what > > > assumptions are being swept under the rug in K1 and K2. > > > > > > As pointed out, 2 adjacent TCP segments should allow the > > > assembly of one FPDU. (In fact it is only the strangest > > > of corner cases where this is not true). So the question > > > becomes, how many TCP segments might have to be buffered > > > before an FPDU is assembled and can be placed? > > > > > > Phrased that way, the answer is obviously half of the > > > TCP segments that can be in flight. The worst case scenario > > > is clearly dropping every other TCP segment. (And as > > > previously noted, if a load-balancing router dispatches > > > successive segments by alternate path that is *exactly* what > > > could happen if one path is delayed). > > > > > > I believe that is a simpler explanation, and formula. > > > It does not change any of the conclusions, however. > > > > > > If the formula presented actually represent a *lower* > > > estimate of the required buffering then the rationale > > > should be explained. > > > > > > Caitlin Bestler > > > http://asomi.com/CaitlinBestler/ > > > _______________________________________________ > > > rddp mailing list > > > rddp@ietf.org > > > https://www1.ietf.org/mailman/listinfo/rddp > >_______________________________________________ > >rddp mailing list > >rddp@ietf.org > >https://www1.ietf.org/mailman/listinfo/rddp > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 12:40:04 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20057 for ; Mon, 24 Feb 2003 12:40:03 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OHmeJ09763 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 12:48:40 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OHmep09760 for ; Mon, 24 Feb 2003 12:48:40 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20051 for ; Mon, 24 Feb 2003 12:39:32 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OHm1p09710; Mon, 24 Feb 2003 12:48:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OHlYp09682 for ; Mon, 24 Feb 2003 12:47:34 -0500 Received: from dmz1.silverbacksystems.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA19997 for ; Mon, 24 Feb 2003 12:38:26 -0500 (EST) Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id 4DF0AD145; Mon, 24 Feb 2003 09:42:07 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id 1007F16C45; Mon, 24 Feb 2003 09:42:07 -0800 (PST) Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii" Date: Mon, 24 Feb 2003 09:42:05 -0800 From: "Somesh Gupta" Importance: Normal In-Reply-To: <5.1.0.14.2.20030224065953.0200bb10@esmail.cup.hp.com> Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 29855-16D436A0; Mon, 24 Feb 2003 09:41:52 -0800 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id 2163D1E6B; Mon, 24 Feb 2003 09:41:52 -0800 (PST) Subject: RE: [rddp] Marker Interval To: "Michael Krause" , X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit The discussion is related to existing software stacks and interfaces. For storage protocols (as is for iSCSI), this forms the first wave of client side implementations. Case in point, even though the knowledge of doing checksum while copying has been around for a long time, how many actually implement it. Somesh > -----Original Message----- > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > Michael Krause > Sent: Monday, February 24, 2003 7:07 AM > To: rddp@ietf.org > Subject: Re: [rddp] Marker Interval > > > At 09:29 AM 2/24/2003 -0500, Black_David@emc.com wrote: > >Somesh Gupta wrote: > > > > > >As David pointed out, a smaller marker > > > >interval puts an unacceptable burden on a software client > > > >(I thing the example was of a 64KB read/write requiring > > > >140+ markers - each of which causes creation of that many > > > >segments, plus creating and inserting markers and marker > > > >vectors etc) - worse than simply using a NIC doing a large > > > >TCP send or TOE offload. > > > >That's definitely a problem. > > > >Caitlin Bestler wrote: > > > > > The pattern where the true payload is a repeating pattern. > > > 140 markers does not require 140 distinct descriptors. > > > If you look up old mini-computers you'll find many examples > > > of repeating groups in DMA control. > > > >That's mostly a statement about building new hardware. A lot of > >existing DMA hardware, OS syscall interfaces, etc. don't support > >repeating descriptors. > > > >Michael Krause wrote: > > > > > The optimal design is a hardware implementation that provides marker > > > insertion / removal. The objective is to insure that one can > implement > > > and interoperate with a software-based solution without creating a > > > complete performance nightmare but also with the understanding that a > > > software-based solution will not perform as well as a hardware-based > > > solution. > > > >The 512-byte fixed marker interval appears to have been over-optimized > >for optimal (new hardware) designs to the detriment of (re)use of > >existing software stacks. The design rationale that I've seen based > >on minimum required IP payload sizes and the desire for a "flow-through" > >implementation is not very convincing, especially as the current MPA > >design can't achieve "flow-through" due to the inability for a receiver > >to determine whether an aligned stream will stay aligned (yes, one > >could build a "flow-through" implementation, but it will have robustness > >problems akin to a TCP implementation with inadequate buffering). In > >addition, we can go back through the "there's no such thing as a buffer- > >less [R]NIC discussion" if that's necessary, but more to the point is > >that the number of descriptors for reasonable-sized operations > >(e.g., 16 or more for a single 8KB transfer) is hard to justify. > > The 512-byte fixed marker interval proposed was based on > requirement to be > compliant with existing RFCs, analysis of packet sizes relative > to targeted > workloads, ability to efficiently implement in hardware, etc. > > Depending upon the software stack implementation, the insertion > of markers > can be accomplished at the same time one is copying / calculating the > checksum for the packet thus it is relatively isolated within the > stack. This was verified by examination of various OSV offerings > including active support by multiple OSVs who stated this was not > an issue > after they had examined their implementations to validate the > proposal and > its cost of implementation. > > The consensus of many involved in the draft development was that due > diligence was done in examining the impacts to software and make > the right > trade-off between the two types of implementations. > > Mike > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 12:45:09 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20301 for ; Mon, 24 Feb 2003 12:45:09 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OHrk809992 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 12:53:46 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OHrkp09989 for ; Mon, 24 Feb 2003 12:53:46 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20294 for ; Mon, 24 Feb 2003 12:44:38 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OHr1p09955; Mon, 24 Feb 2003 12:53:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OHqbp09939 for ; Mon, 24 Feb 2003 12:52:37 -0500 Received: from dmz1.silverbacksystems.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20254 for ; Mon, 24 Feb 2003 12:43:29 -0500 (EST) Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id C20C5D14E; Mon, 24 Feb 2003 09:47:09 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id 7D9DA197A8; Mon, 24 Feb 2003 09:47:09 -0800 (PST) Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="iso-8859-1" Date: Mon, 24 Feb 2003 09:47:05 -0800 From: "Somesh Gupta" Importance: Normal In-Reply-To: <01CA1B1F74A4EC40A9C887F7B58F8724031430DE@cceexc16.americas.cpqcorp.net> Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 30069-7579EBDC; Mon, 24 Feb 2003 09:46:52 -0800 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id 69B99196EB; Mon, 24 Feb 2003 09:46:52 -0800 (PST) Subject: RE: [rddp] Marker Interval To: "Barron, Dwight" , , X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Dwight, The point is of allowing the marker interval to be flexible so that hardware implementations that want to work well with software based senders may allow a larger marker interval or not require it at all. Somesh > -----Original Message----- > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > Barron, Dwight > Sent: Monday, February 24, 2003 7:56 AM > To: Black_David@emc.com; rddp@ietf.org > Subject: RE: [rddp] Marker Interval > > > Assuming we had a large number of negotiable marker intervals, > what would the optimum negotiation strategy be for a software-MPA > to connect to a hardware-MPA implementation? Clearly, the SW-MPA > would want larger intervals to minimize overhead, while > presumably the HW-MPA would want smaller intervals to minimize > buffering. While I see a lot of value in having mixed mode > connections, I keep coming back the notion that the SW > implementation is willingly taking on additional overhead for the > benefit of the HW implementation, in which case the SW end would > accept the HW's recommended marker interval value. > > Regards, > Dwight > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Monday, February 24, 2003 8:30 AM > > To: rddp@ietf.org > > Subject: [rddp] Marker Interval > > > > > > Somesh Gupta wrote: > > > > > >As David pointed out, a smaller marker > > > >interval puts an unacceptable burden on a software client > > > >(I thing the example was of a 64KB read/write requiring > > > >140+ markers - each of which causes creation of that many > > > >segments, plus creating and inserting markers and marker > > > >vectors etc) - worse than simply using a NIC doing a large > > > >TCP send or TOE offload. > > > > That's definitely a problem. > > > > Caitlin Bestler wrote: > > > > > The pattern where the true payload is a repeating pattern. > > > 140 markers does not require 140 distinct descriptors. > > > If you look up old mini-computers you'll find many examples > > > of repeating groups in DMA control. > > > > That's mostly a statement about building new hardware. A lot of > > existing DMA hardware, OS syscall interfaces, etc. don't support > > repeating descriptors. > > > > Michael Krause wrote: > > > > > The optimal design is a hardware implementation that > > provides marker > > > insertion / removal. The objective is to insure that one > > can implement > > > and interoperate with a software-based solution without creating a > > > complete performance nightmare but also with the > > understanding that a > > > software-based solution will not perform as well as a hardware-based > > > solution. > > > > The 512-byte fixed marker interval appears to have been over-optimized > > for optimal (new hardware) designs to the detriment of (re)use of > > existing software stacks. The design rationale that I've seen based > > on minimum required IP payload sizes and the desire for a > > "flow-through" > > implementation is not very convincing, especially as the current MPA > > design can't achieve "flow-through" due to the inability for > > a receiver > > to determine whether an aligned stream will stay aligned (yes, one > > could build a "flow-through" implementation, but it will have > > robustness > > problems akin to a TCP implementation with inadequate buffering). In > > addition, we can go back through the "there's no such thing > > as a buffer- > > less [R]NIC discussion" if that's necessary, but more to the point is > > that the number of descriptors for reasonable-sized operations > > (e.g., 16 or more for a single 8KB transfer) is hard to justify. > > > > Thanks, > > --David > > ---------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 176 South St., Hopkinton, MA 01748 > > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > ---------------------------------------------------- > > _______________________________________________ > > rddp mailing list > > rddp@ietf.org > > https://www1.ietf.org/mailman/listinfo/rddp > > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 12:53:11 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20570 for ; Mon, 24 Feb 2003 12:53:11 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OI1mE10379 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 13:01:48 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OI1lp10376 for ; Mon, 24 Feb 2003 13:01:47 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20546 for ; Mon, 24 Feb 2003 12:52:39 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OI12p10322; Mon, 24 Feb 2003 13:01:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OI0Zp10284 for ; Mon, 24 Feb 2003 13:00:35 -0500 Received: from palrel13.hp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20499 for ; Mon, 24 Feb 2003 12:51:26 -0500 (EST) Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel13.hp.com (Postfix) with ESMTP id 11D5D1C00B8B for ; Mon, 24 Feb 2003 09:55:19 -0800 (PST) Received: from mk731915.cup.hp.com ([15.113.161.2]) by esmail.cup.hp.com (8.9.3/8.8.6) with ESMTP id JAA04156 for ; Mon, 24 Feb 2003 09:52:38 -0800 (PST) Message-Id: <5.1.0.14.2.20030224092427.03277ce0@esmail.cup.hp.com> X-Sender: krause@esmail.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Mon, 24 Feb 2003 09:39:42 -0800 To: rddp@ietf.org From: Michael Krause In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C923@corpmx14.us.dg.co m> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Subject: [rddp] Re: Markers and Software Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 12:16 PM 2/24/2003 -0500, Black_David@emc.com wrote: > > Depending upon the software stack implementation, the insertion of markers > > > can be accomplished at the same time one is copying / calculating the > > checksum for the packet thus it is relatively isolated within the > > stack. This was verified by examination of various OSV offerings > > including active support by multiple OSVs who stated this was not an issue > > > after they had examined their implementations to validate the proposal and > > > its cost of implementation. > >That is software, but not an unmodified existing stack, which is of >importance here, in contrast to the environment in which the MPA draft >was originally developed. I think we need to keep in mind that there is rarely any such thing as a pure layered network stack implementation in practice. Most software implementations today use a variety of techniques to provide performance boosts to their solution offerings and do so while maintaining interoperability. Some simple ones that come to mind are: - Copy and checksum when crossing the user-kernel boundary - Buffer cache (file system) accelerations for network accessed objects - Checksum off-load - TCP large send (send a single large message to a NIC and have it segment accordingly). If the various OSV want to isolate this functionality as they currently do today and they want to add in MPA through similar means, no one will be able to tell the difference from the wire protocol perspective and they will interoperate just fine as well as perform better than those that do not take these types of steps. If an OSV wants to have a well-layered implementation, then it can do so without changing its TCP implementation. The layer above TCP (e.g. Sockets) simply performs the marker insertion (transparent to the application) prior to handing the packet to the TCP layer which treats it as a standard byte stream for transmission purposes. The performance impacts can be mitigated for those that want to implement a higher performing solution with some intelligence such as utilizing various techniques they may have already in place. We should also keep in mind that negotiating attributes is generally more disruptive to an OSV / network stack implementation than having a single method for accomplishing a task - a key reason for having a fixed marker interval. In general, I have not seen a strong case made for having variable marker intervals or necessarily an implementation concern being raised by various OSVs (the OS we provide within my company do not have an issue with the current proposal) w.r.t. complexity, performance, etc. Mike _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 13:02:12 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA20873 for ; Mon, 24 Feb 2003 13:02:12 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OIAnX11454 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 13:10:49 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OIAnp11451 for ; Mon, 24 Feb 2003 13:10:49 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA20836 for ; Mon, 24 Feb 2003 13:01:41 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OIA6p11434; Mon, 24 Feb 2003 13:10:06 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OI9Ip11387 for ; Mon, 24 Feb 2003 13:09:18 -0500 Received: from palrel11.hp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA20758 for ; Mon, 24 Feb 2003 13:00:10 -0500 (EST) Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel11.hp.com (Postfix) with ESMTP id BCB211C0163F for ; Mon, 24 Feb 2003 10:04:03 -0800 (PST) Received: from mk731915.cup.hp.com ([15.113.161.2]) by esmail.cup.hp.com (8.9.3/8.8.6) with ESMTP id KAA04715 for ; Mon, 24 Feb 2003 10:01:22 -0800 (PST) Message-Id: <5.1.0.14.2.20030224095746.032abe70@esmail.cup.hp.com> X-Sender: krause@esmail.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Mon, 24 Feb 2003 10:04:00 -0800 To: From: Michael Krause Subject: RE: [rddp] Marker Interval In-Reply-To: References: <5.1.0.14.2.20030224065953.0200bb10@esmail.cup.hp.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 09:42 AM 2/24/2003 -0800, Somesh Gupta wrote: >The discussion is related to existing software stacks and >interfaces. For storage protocols (as is for iSCSI), this >forms the first wave of client side implementations. Again this is a function of implementation and a well-layered implementation can be delivered without requiring any network stack modifications if desired. Same applies to software based iSCSI which has been demonstrated by multiple vendors without network stack modifications. Some of these implementations now deliver sufficient performance that current customers using them are not raising any major objections. Given this can be done with iSCSI and markers are clearly less complex to implement / support in software than iSCSI, one might argue that the complexity / cost issues are not as high as some assert. >Case in point, even though the knowledge of doing checksum >while copying has been around for a long time, how many >actually implement it. A lot of the Unix implementations I'm familiar with clearly do. I'll let Windows speak for itself but it isn't hard and has been done for quite some time and I don't see any issue with adding in a marker bit of code to the mix. Same will apply to adding in SCTP's CRC for those implementations using SCTP today. The key is that this is all implementation specific and there are clear demonstrations of shipping product showing that the complexity / performance cost is not high and thus a reasonable balance is available with MPA / DDP / RDMA as currently proposed in the draft specifications. Mike _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 13:12:04 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA21184 for ; Mon, 24 Feb 2003 13:12:04 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OIKfk11885 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 13:20:41 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OIKfp11880 for ; Mon, 24 Feb 2003 13:20:41 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA21157 for ; Mon, 24 Feb 2003 13:11:33 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OIK1p11789; Mon, 24 Feb 2003 13:20:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OIICp11708 for ; Mon, 24 Feb 2003 13:18:12 -0500 Received: from mxic2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA21055 for ; Mon, 24 Feb 2003 13:09:04 -0500 (EST) From: Black_David@emc.com Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRHVR0T>; Mon, 24 Feb 2003 13:12:57 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C925@corpmx14.us.dg.com> To: krause@cup.hp.com, rddp@ietf.org Subject: RE: [rddp] Re: Markers and Software Date: Mon, 24 Feb 2003 13:10:25 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > At 12:16 PM 2/24/2003 -0500, Black_David@emc.com wrote: > > > Depending upon the software stack implementation, the insertion of markers > > > can be accomplished at the same time one is copying / calculating the > > > checksum for the packet thus it is relatively isolated within the > > > stack. This was verified by examination of various OSV offerings > > > including active support by multiple OSVs who stated this was not an issue > > > after they had examined their implementations to validate the proposal and > > > its cost of implementation. > > > >That is software, but not an unmodified existing stack, which is of > >importance here, in contrast to the environment in which the MPA draft > >was originally developed. > > > I think we need to keep in mind that there is rarely any such thing as a > pure layered network stack implementation in practice. [ ... large snip ...] > In general, I have not seen a strong case made for having variable marker > intervals or necessarily an implementation concern being raised by various > OSVs (the OS we provide within my company do not have an issue with the > current proposal) w.r.t. complexity, performance, etc. That is based on the incorrect assumption that the only interesting software implementations are by Operating Systems Vendors who will change their stacks to do the implementation. Not only is that a bad assumption, but it does not comport with the current WG rough consensus of the WG that support for at least client implementations on existing unmodified stacks is important. It's ok not to optimize for unmodified existing stacks, but the design cannot deliberately ignore them in the above fashion. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 13:40:35 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20571 for ; Mon, 24 Feb 2003 12:53:11 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OI1m410395 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 13:01:48 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OI1mp10392 for ; Mon, 24 Feb 2003 13:01:48 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20548 for ; Mon, 24 Feb 2003 12:52:39 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OI13p10338; Mon, 24 Feb 2003 13:01:03 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OI0ap10288 for ; Mon, 24 Feb 2003 13:00:36 -0500 Received: from palrel10.hp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA20501 for ; Mon, 24 Feb 2003 12:51:27 -0500 (EST) Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel10.hp.com (Postfix) with ESMTP id A787B1C0186F for ; Mon, 24 Feb 2003 09:55:20 -0800 (PST) Received: from mk731915.cup.hp.com ([15.113.161.2]) by esmail.cup.hp.com (8.9.3/8.8.6) with ESMTP id JAA04159 for ; Mon, 24 Feb 2003 09:52:38 -0800 (PST) Message-Id: <5.1.0.14.2.20030224095352.03236500@esmail.cup.hp.com> X-Sender: krause@esmail.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Mon, 24 Feb 2003 09:53:54 -0800 To: From: Michael Krause Subject: RE: [rddp] MPA TCP analysis draft Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 09:24 AM 2/24/2003 -0800, Somesh Gupta wrote: > > Implementations clearly do not have to buffer any headers - they > > only need > > to understand the number of in-flight messages and know when the > > LLP window > > has been filled in such that a given message can be delivered. > > Tracking of > > such information can be done, for example, within the posted work request > > which is used by the implementation to describe the associated physical > > buffer if desired. This is again an implementation option, and > > as one can > > see, it is fairly straight forward to implement in hardware - there are > > many other implementation choices one could use to solve this type of > > problem without requiring the headers to be maintained. > > Clearly some form of information has to be kept (does not have to > be headers). If the DDP rules are to be verified, then more information > has to be kept. Even if rules are not verified in out of order > segments case, the headers or variation thereof that generate completion > must be kept. The DDP specification should be changed if rules are > not to be verified - otherwise the information to be maintained is > of the order of number of connections. Or scoreboarding can be done, > which is not likely to be easier. The DDP specification is quite clear on how to validate an operation and delineating the responsibilities of the LLP and the DDP layers. How an implementation chooses to enforce these rules is up to it but I do not see anything stated as yet to indicate that the rules are incorrect or that it is not possible to implement a compliant solution (either in software or hardware). > > The optimal design is a hardware implementation that provides marker > > insertion / removal. The objective is to insure that one can > > implement and > > interoperate with a software-based solution without creating a complete > > performance nightmare but also with the understanding that a > > software-based > > solution will not perform as well as a hardware-based solution. The > > current draft specifications attempt to strike a good balance in allowing > > high-performance hardware implementations while insuring > > software-based can > > be constructed. I know of at least a couple of software implementations > > that are in-flight today and no one is stating the use of markers or its > > proposed interval are a problem. > > I have to disagree with both points over here. Markers do pose a > burden on software implementations (vectors have to be split, > memory for markers has to be allocated and vectors created for it, > number of calls to system have to increase, a number of completions > from NIC will increase. The penalty is quite significant. I have a different implementation in mind here so obviously costs will vary. A minimum implementation in software will have: - If the data + markers fits into a single segment, then there is no additional memory costs as most implementations use power of 2 memory buffers to simplify the design so finding the right size with some additional room may be nothing more than moving up to the next power of 2 buffer size. - If the data + markers does not fit into a single segment, then it will require an additional segment for the overflow caused by markers. However, most software implementations already deal with packet trains today so the overhead is not significant (function of implementation). - The number of calls to insert / remove markers is worse case 1 additional call but most implementations will perform these operations as part of their copy or checksum calculations. - There is no increase in the number of completion events from the NIC except perhaps when there are additional packets received. Most good NIC implementations already contain coalescence techniques to reduce / eliminate completion events wherever possible so again this may be implementation specific. From our work to date, the implementation complexity is not high nor excessively burdensome; mileage will vary. > Similarly, the current draft does not allow optimization of RNICs > that work well with software based senders (or penalizes it to > the point that it is no longer a very viable choice). The IFT draft > strikes a much better balance for both environments. Please specify. I have yet to find a RNIC that creates any type of penalty regarding a software-based implementation as the RNIC is implementing the RDMA / DDP components already. Now, if you are referring to a TOE interface, then this is a function of the TOE implementation as to what it provides. A well-layered implementation would simply hand a buffer with the markers already included to the TOE interface who would transmit the buffer just like any other traffic. For inbound, the TOE would deliver a buffer normally and it would be up to the From our prototype work to date with TOE, we have not found any undue complexity / penalties and this is what I expect given the MPA / DDP / RDMA specifications are capable of being implemented over a well-layered implementation. Mike _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 13:47:06 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA22312 for ; Mon, 24 Feb 2003 13:47:06 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OItiT14258 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 13:55:44 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OItip14255 for ; Mon, 24 Feb 2003 13:55:44 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA22301 for ; Mon, 24 Feb 2003 13:46:35 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OIt5p14212; Mon, 24 Feb 2003 13:55:05 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OIsBp14164 for ; Mon, 24 Feb 2003 13:54:11 -0500 Received: from mxic1.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA22217 for ; Mon, 24 Feb 2003 13:45:02 -0500 (EST) From: Black_David@emc.com Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1KDZDK56>; Mon, 24 Feb 2003 13:48:13 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C928@corpmx14.us.dg.com> To: krause@cup.hp.com, rddp@ietf.org Subject: RE: [rddp] Marker Interval Date: Mon, 24 Feb 2003 13:46:22 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > Again this is a function of implementation and a well-layered > implementation can be delivered without requiring any network stack > modifications if desired. Same applies to software based iSCSI which has > been demonstrated by multiple vendors without network stack > modifications. Some of these implementations now deliver sufficient > performance that current customers using them are not raising any major > objections. Given this can be done with iSCSI and markers are clearly less > complex to implement / support in software than iSCSI, one might argue that > the complexity / cost issues are not as high as some assert. Unless those iSCSI implementations include iSCSI markers at a 512-byte interval (which I very much doubt), this sort of comparison is apples vs. oranges, as in the absence of markers, those implementations can send a 4K, 8K or larger data block in one operation. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 14:09:05 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22933 for ; Mon, 24 Feb 2003 14:09:04 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OJHh615857 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 14:17:43 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OJHhp15854 for ; Mon, 24 Feb 2003 14:17:43 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22889 for ; Mon, 24 Feb 2003 14:08:33 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OJH3p15783; Mon, 24 Feb 2003 14:17:03 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OJGKp15666 for ; Mon, 24 Feb 2003 14:16:20 -0500 Received: from dmz1.silverbacksystems.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA22878 for ; Mon, 24 Feb 2003 14:07:10 -0500 (EST) Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id 41EB7D16B; Mon, 24 Feb 2003 11:10:51 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id 03540882F; Mon, 24 Feb 2003 11:10:51 -0800 (PST) Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii" Date: Mon, 24 Feb 2003 11:10:25 -0800 From: "Somesh Gupta" Importance: Normal In-Reply-To: <5.1.0.14.2.20030224095746.032abe70@esmail.cup.hp.com> Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 00890-146373CC; Mon, 24 Feb 2003 11:10:12 -0800 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id A2E2D195BE; Mon, 24 Feb 2003 11:10:12 -0800 (PST) Subject: RE: [rddp] Marker Interval To: "Michael Krause" , X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit > -----Original Message----- > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > Michael Krause > Sent: Monday, February 24, 2003 10:04 AM > To: rddp@ietf.org > Subject: RE: [rddp] Marker Interval > > > At 09:42 AM 2/24/2003 -0800, Somesh Gupta wrote: > >The discussion is related to existing software stacks and > >interfaces. For storage protocols (as is for iSCSI), this > >forms the first wave of client side implementations. > > Again this is a function of implementation and a well-layered > implementation can be delivered without requiring any network stack > modifications if desired. Same applies to software based iSCSI which has > been demonstrated by multiple vendors without network stack > modifications. Some of these implementations now deliver sufficient > performance that current customers using them are not raising any major > objections. Given this can be done with iSCSI and markers are > clearly less > complex to implement / support in software than iSCSI, one might > argue that > the complexity / cost issues are not as high as some assert. I don't know what the point here. You can run a distributed database over a software implementation which is clearly less complex than markers. How does that imply that the cost of markers is not high? In terms of complexity, it is close to 0 complexity to implement markers in software (especially when you don't have to modify the stack to do so). > > >Case in point, even though the knowledge of doing checksum > >while copying has been around for a long time, how many > >actually implement it. > > A lot of the Unix implementations I'm familiar with clearly do. I'll let > Windows speak for itself but it isn't hard and has been done for > quite some > time and I don't see any issue with adding in a marker bit of code to the > mix. Same will apply to adding in SCTP's CRC for those implementations > using SCTP today. The key is that this is all implementation > specific and > there are clear demonstrations of shipping product showing that the > complexity / performance cost is not high and thus a reasonable > balance is > available with MPA / DDP / RDMA as currently proposed in the draft > specifications. > > Mike > > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 14:14:07 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA23146 for ; Mon, 24 Feb 2003 14:14:07 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OJMkI16419 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 14:22:46 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OJMkp16416 for ; Mon, 24 Feb 2003 14:22:46 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA23129 for ; Mon, 24 Feb 2003 14:13:36 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OJM1p16387; Mon, 24 Feb 2003 14:22:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OJLqp16372 for ; Mon, 24 Feb 2003 14:21:52 -0500 Received: from palrel13.hp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA23084 for ; Mon, 24 Feb 2003 14:12:41 -0500 (EST) Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel13.hp.com (Postfix) with ESMTP id E3F481C00BE9 for ; Mon, 24 Feb 2003 11:16:23 -0800 (PST) Received: from mk731915.cup.hp.com ([15.113.161.2]) by esmail.cup.hp.com (8.9.3/8.8.6) with ESMTP id LAA05208 for ; Mon, 24 Feb 2003 11:13:39 -0800 (PST) Message-Id: <5.1.0.14.2.20030224110003.032b6eb0@esmail.cup.hp.com> X-Sender: krause@esmail.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Mon, 24 Feb 2003 11:04:40 -0800 To: rddp@ietf.org From: Michael Krause Subject: RE: [rddp] Re: Markers and Software In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C925@corpmx14.us.dg.co m> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 01:10 PM 2/24/2003 -0500, Black_David@emc.com wrote: > > At 12:16 PM 2/24/2003 -0500, Black_David@emc.com wrote: > > > > Depending upon the software stack implementation, the insertion of >markers > > > > can be accomplished at the same time one is copying / calculating the > > > > checksum for the packet thus it is relatively isolated within the > > > > stack. This was verified by examination of various OSV offerings > > > > including active support by multiple OSVs who stated this was not an >issue > > > > after they had examined their implementations to validate the proposal >and > > > > its cost of implementation. > > > > > >That is software, but not an unmodified existing stack, which is of > > >importance here, in contrast to the environment in which the MPA draft > > >was originally developed. > > > > > > I think we need to keep in mind that there is rarely any such thing as a > > pure layered network stack implementation in practice. > >[ ... large snip ...] > > > In general, I have not seen a strong case made for having variable marker > > intervals or necessarily an implementation concern being raised by various > > > OSVs (the OS we provide within my company do not have an issue with the > > current proposal) w.r.t. complexity, performance, etc. > >That is based on the incorrect assumption that the only interesting >software implementations are by Operating Systems Vendors who will >change their stacks to do the implementation. Not only is that a >bad assumption, but it does not comport with the current WG rough >consensus of the WG that support for at least client implementations >on existing unmodified stacks is important. It's ok not to optimize >for unmodified existing stacks, but the design cannot deliberately >ignore them in the above fashion. I did not state that either clients or storage are being ignored nor did I state that the only interesting implementations are by OSV who will change their stacks. I only stated that the one can implement without changing that stack and that a more optimal implementation can be done by providing minor changes. This isn't apple vs. oranges but illustrates my point that there are many shades of implementation and mileage will vary accordingly. Does the architecture need to be permuted to accommodate every implementation or can it strike a good balance - I assert that their is a good balance in today's drafts for all solutions and I have not seen anything substantial to the contrary. Mike _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 14:35:07 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA23893 for ; Mon, 24 Feb 2003 14:35:07 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OJhkP18359 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 14:43:46 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OJhkp18356 for ; Mon, 24 Feb 2003 14:43:46 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA23867 for ; Mon, 24 Feb 2003 14:34:36 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OJh2p18297; Mon, 24 Feb 2003 14:43:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OJgQp18235 for ; Mon, 24 Feb 2003 14:42:26 -0500 Received: from hermes.fm.intel.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA23800 for ; Mon, 24 Feb 2003 14:33:15 -0500 (EST) Received: from petasus.fm.intel.com (petasus.fm.intel.com [10.1.192.37]) by hermes.fm.intel.com (8.11.6/8.11.6/d: outer.mc,v 1.51 2002/09/23 20:43:23 dmccart Exp $) with ESMTP id h1OJY0I26642 for ; Mon, 24 Feb 2003 19:34:00 GMT Received: from fmsmsxvs042.fm.intel.com (fmsmsxvs042.fm.intel.com [132.233.42.128]) by petasus.fm.intel.com (8.11.6/8.11.6/d: inner.mc,v 1.28 2003/01/13 19:44:39 dmccart Exp $) with SMTP id h1OJVib21574 for ; Mon, 24 Feb 2003 19:31:44 GMT Received: from fmsmsx29.FM.INTEL.COM ([132.233.42.29]) by fmsmsxvs042.fm.intel.com (NAVGW 2.5.2.11) with SMTP id M2003022411393917427 ; Mon, 24 Feb 2003 11:39:39 -0800 Received: by fmsmsx29.fm.intel.com with Internet Mail Service (5.5.2653.19) id ; Mon, 24 Feb 2003 11:37:07 -0800 Message-ID: From: "Larsen, Roy K" To: "'Michael Krause'" , rddp@ietf.org Subject: RE: [rddp] Marker Interval Date: Mon, 24 Feb 2003 11:37:06 -0800 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > -----Original Message----- > From: Michael Krause [mailto:krause@cup.hp.com] > > At 09:42 AM 2/24/2003 -0800, Somesh Gupta wrote: > >The discussion is related to existing software stacks and > >interfaces. For storage protocols (as is for iSCSI), this > >forms the first wave of client side implementations. > > Again this is a function of implementation and a well-layered > implementation can be delivered without requiring any network stack > modifications if desired. Same applies to software based > iSCSI which has > been demonstrated by multiple vendors without network stack > modifications. Some of these implementations now deliver sufficient > performance that current customers using them are not raising > any major > objections. Given this can be done with iSCSI and markers > are clearly less > complex to implement / support in software than iSCSI, one > might argue that > the complexity / cost issues are not as high as some assert. Mike, Marker use in iSCSI is a negotiated option. Do you know if these successful software iSCSI implementations are running with markers enabled? Roy _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 15:10:09 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA24951 for ; Mon, 24 Feb 2003 15:10:09 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OKIoZ20412 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 15:18:50 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OKInp20409 for ; Mon, 24 Feb 2003 15:18:49 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA24914 for ; Mon, 24 Feb 2003 15:09:38 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OKI5p20359; Mon, 24 Feb 2003 15:18:05 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OKD2p20146 for ; Mon, 24 Feb 2003 15:13:02 -0500 Received: from palrel10.hp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA24619 for ; Mon, 24 Feb 2003 15:03:50 -0500 (EST) Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel10.hp.com (Postfix) with ESMTP id 70C1D1C01188 for ; Mon, 24 Feb 2003 12:07:43 -0800 (PST) Received: from mk731915.cup.hp.com ([15.113.161.2]) by esmail.cup.hp.com (8.9.3/8.8.6) with ESMTP id MAA08885 for ; Mon, 24 Feb 2003 12:05:02 -0800 (PST) Message-Id: <5.1.0.14.2.20030224113354.0200fed8@esmail.cup.hp.com> X-Sender: krause@esmail.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Mon, 24 Feb 2003 12:07:36 -0800 To: From: Michael Krause Subject: RE: [rddp] Marker Interval In-Reply-To: References: <5.1.0.14.2.20030224095746.032abe70@esmail.cup.hp.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 11:10 AM 2/24/2003 -0800, Somesh Gupta wrote: > > -----Original Message----- > > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > > Michael Krause > > Sent: Monday, February 24, 2003 10:04 AM > > To: rddp@ietf.org > > Subject: RE: [rddp] Marker Interval > > > > > > At 09:42 AM 2/24/2003 -0800, Somesh Gupta wrote: > > >The discussion is related to existing software stacks and > > >interfaces. For storage protocols (as is for iSCSI), this > > >forms the first wave of client side implementations. > > > > Again this is a function of implementation and a well-layered > > implementation can be delivered without requiring any network stack > > modifications if desired. Same applies to software based iSCSI which has > > been demonstrated by multiple vendors without network stack > > modifications. Some of these implementations now deliver sufficient > > performance that current customers using them are not raising any major > > objections. Given this can be done with iSCSI and markers are > > clearly less > > complex to implement / support in software than iSCSI, one might > > argue that > > the complexity / cost issues are not as high as some assert. > > I don't know what the point here. - That markers are not complex to implement nor are they that major of a performance loss as the data is already being moved from one buffer to another since the hardware could not perform the direct placement when using a software stack. - That markers can be implemented on top of an existing network stack keeping things well layered which is one of the objectives being stated on this e-mail string. >You can run a distributed database over a software implementation which is >clearly less complex > than markers. How does that imply that the cost of markers is not high? I've requested a definition of "high" as the discussion to date is perhaps too subjective. I described a potential implementation to illustrate what can be done that IMO is not complex or a major performance inhibitor. >In terms of complexity, it is close to 0 complexity to implement markers >in software (especially when you don't have to modify the stack to do so). Then we are in agreement that the complexity is not high in software for a well-layered implementation. Is there any remaining disagreement in this regard? Mike _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 16:26:06 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA27066 for ; Mon, 24 Feb 2003 16:26:06 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OLYnA25488 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 16:34:49 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OLYmp25485 for ; Mon, 24 Feb 2003 16:34:48 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA27048 for ; Mon, 24 Feb 2003 16:25:34 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OLY6p25457; Mon, 24 Feb 2003 16:34:06 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OLXgp25443 for ; Mon, 24 Feb 2003 16:33:42 -0500 Received: from ztxmail04.ztx.compaq.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id QAA27038 for ; Mon, 24 Feb 2003 16:24:28 -0500 (EST) Received: from cceexg13.americas.cpqcorp.net (cceexg13.americas.cpqcorp.net [16.110.250.119]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id F05D34F1F; Mon, 24 Feb 2003 15:28:21 -0600 (CST) Received: from cceexc16.americas.cpqcorp.net ([16.110.250.58]) by cceexg13.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Mon, 24 Feb 2003 15:28:21 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: RE: [rddp] Marker Interval Date: Mon, 24 Feb 2003 15:28:20 -0600 Message-ID: <01CA1B1F74A4EC40A9C887F7B58F8724031430DF@cceexc16.americas.cpqcorp.net> Thread-Topic: [rddp] Marker Interval Thread-Index: AcLcLNC1S4BzJH2bScSDoxEDg6wqZAAFSglg From: "Barron, Dwight" To: "Somesh Gupta" , X-OriginalArrivalTime: 24 Feb 2003 21:28:21.0666 (UTC) FILETIME=[A7DBE820:01C2DC4B] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h1OLXgp25444 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit There are clearly a wide spectrum of implementation possibilities, but I like to focus on the endpoints first, then determine how many additional click-stops are necessary to adequately support the middle of the spectrum. Clearly, support of an ULP that wants to invoke RDDP over an unmodified TCP stack is an important goal, but optionality isn't free, so let's examine what we are really buying with negotiable marker intervals. I'll assume that the ULP using RDDP wants to see RDDP semantics (whatever that may be) and not interim RDDP headers and components. If you assume this RDDP "library" lives on top of unmodified sockets/TCP, then I believe there is no choice but for a receive stream to a) sequentially read every byte for purposes of CRC calculation and b) copy incoming stream data to separate the RDDP headers from ULP payload. This is especially true of tagged messages where the ULP is using a contiguous memory address range to reference data. On the transmit side, the R! DDP library will still have to touch every byte for CRC calculation purposes, but may try to avoid copying by using multiple send calls. But at what point does copying become cheaper than multiple send calls? Especially if the sender is at least *attempting* to provide header alignment, which would mean adding RDDP headers for each TCP_MAXSEG interval. While we all like to avoid data copies (the whole reason for RDDP to begin with), once we bite the bullet for software CRCs (which is clearly the case with standard sockets), the copy really isn't that painful. Regards, Dwight > -----Original Message----- > From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com] > Sent: Monday, February 24, 2003 11:47 AM > To: Barron, Dwight; Black_David@emc.com; rddp@ietf.org > Subject: RE: [rddp] Marker Interval > > > Dwight, > > The point is of allowing the marker interval to be flexible > so that hardware implementations that want to work well with > software based senders may allow a larger marker interval > or not require it at all. > > Somesh > > > -----Original Message----- > > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > > Barron, Dwight > > Sent: Monday, February 24, 2003 7:56 AM > > To: Black_David@emc.com; rddp@ietf.org > > Subject: RE: [rddp] Marker Interval > > > > > > Assuming we had a large number of negotiable marker intervals, > > what would the optimum negotiation strategy be for a software-MPA > > to connect to a hardware-MPA implementation? Clearly, the SW-MPA > > would want larger intervals to minimize overhead, while > > presumably the HW-MPA would want smaller intervals to minimize > > buffering. While I see a lot of value in having mixed mode > > connections, I keep coming back the notion that the SW > > implementation is willingly taking on additional overhead for the > > benefit of the HW implementation, in which case the SW end would > > accept the HW's recommended marker interval value. > > > > Regards, > > Dwight > > > > > -----Original Message----- > > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > > Sent: Monday, February 24, 2003 8:30 AM > > > To: rddp@ietf.org > > > Subject: [rddp] Marker Interval > > > > > > > > > Somesh Gupta wrote: > > > > > > > >As David pointed out, a smaller marker > > > > >interval puts an unacceptable burden on a software client > > > > >(I thing the example was of a 64KB read/write requiring > > > > >140+ markers - each of which causes creation of that many > > > > >segments, plus creating and inserting markers and marker > > > > >vectors etc) - worse than simply using a NIC doing a large > > > > >TCP send or TOE offload. > > > > > > That's definitely a problem. > > > > > > Caitlin Bestler wrote: > > > > > > > The pattern where the true payload is a repeating pattern. > > > > 140 markers does not require 140 distinct descriptors. > > > > If you look up old mini-computers you'll find many examples > > > > of repeating groups in DMA control. > > > > > > That's mostly a statement about building new hardware. A lot of > > > existing DMA hardware, OS syscall interfaces, etc. don't support > > > repeating descriptors. > > > > > > Michael Krause wrote: > > > > > > > The optimal design is a hardware implementation that > > > provides marker > > > > insertion / removal. The objective is to insure that one > > > can implement > > > > and interoperate with a software-based solution without > creating a > > > > complete performance nightmare but also with the > > > understanding that a > > > > software-based solution will not perform as well as a > hardware-based > > > > solution. > > > > > > The 512-byte fixed marker interval appears to have been > over-optimized > > > for optimal (new hardware) designs to the detriment of (re)use of > > > existing software stacks. The design rationale that I've > seen based > > > on minimum required IP payload sizes and the desire for a > > > "flow-through" > > > implementation is not very convincing, especially as the > current MPA > > > design can't achieve "flow-through" due to the inability for > > > a receiver > > > to determine whether an aligned stream will stay aligned (yes, one > > > could build a "flow-through" implementation, but it will have > > > robustness > > > problems akin to a TCP implementation with inadequate > buffering). In > > > addition, we can go back through the "there's no such thing > > > as a buffer- > > > less [R]NIC discussion" if that's necessary, but more to > the point is > > > that the number of descriptors for reasonable-sized operations > > > (e.g., 16 or more for a single 8KB transfer) is hard to justify. > > > > > > Thanks, > > > --David > > > ---------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 176 South St., Hopkinton, MA 01748 > > > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > ---------------------------------------------------- > > > _______________________________________________ > > > rddp mailing list > > > rddp@ietf.org > > > https://www1.ietf.org/mailman/listinfo/rddp > > > > > _______________________________________________ > > rddp mailing list > > rddp@ietf.org > > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 17:21:00 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28549 for ; Mon, 24 Feb 2003 17:21:00 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OMThC29667 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 17:29:43 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OMThp29664 for ; Mon, 24 Feb 2003 17:29:43 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28537 for ; Mon, 24 Feb 2003 17:20:28 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OMT3p29650; Mon, 24 Feb 2003 17:29:03 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OMSPp29628 for ; Mon, 24 Feb 2003 17:28:25 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28509 for ; Mon, 24 Feb 2003 17:19:10 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id OAA22563 for ; Mon, 24 Feb 2003 14:23:04 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id <105KW7X7>; Mon, 24 Feb 2003 17:23:04 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B9000@xbl.ma.emulex.com> From: "Williams, Jim" To: rddp@ietf.org Subject: Optional CRCs ( was [rddp] Marker Interval ) Date: Mon, 24 Feb 2003 17:23:03 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > -----Original Message----- > From: Barron, Dwight [mailto:Dwight.Barron@hp.com] > Sent: Monday, February 24, 2003 4:28 PM > To: Somesh Gupta; rddp@ietf.org > Subject: RE: [rddp] Marker Interval > > > There are clearly a wide spectrum of implementation > possibilities, but I like to focus on the endpoints first, > then determine how many additional click-stops are necessary > to adequately support the middle of the spectrum. Clearly, > support of an ULP that wants to invoke RDDP over an > unmodified TCP stack is an important goal, but optionality > isn't free, so let's examine what we are really buying with > negotiable marker intervals. I'll assume that the ULP using > RDDP wants to see RDDP semantics (whatever that may be) and > not interim RDDP headers and components. If you assume this > RDDP "library" lives on top of unmodified sockets/TCP, then I > believe there is no choice but for a receive stream to a) > sequentially read every byte for purposes of CRC calculation > and b) copy incoming stream data to separate the RDDP headers > from ULP payload. This is especially true of tagged messages > where the ULP is using a contiguous memory address range to > reference data. On the transmit side, the R! > DDP library will still have to touch every byte for CRC > calculation purposes, I think if we are serious about supporting unmodified TCP, then the CRC should be optional. NFS typically runs with only TCP/UDP checksum for protection. Not a good idea to do this over the public internet, but then critical data traveling through the public internet will (or should) use IPSec protection. iSCSI is a precedent for optional CRCs. Data errors in iSCSI or NFS are equally catastrophic as RDDP. Tradeoff can be made by user as to required protection, quality and reliability of network infrastructure, etc., to decide whether to use CRCs. > but may try to avoid copying by using > multiple send calls. But at what point does copying become > cheaper than multiple send calls? Especially if the sender is > at least *attempting* to provide header alignment, which > would mean adding RDDP headers for each TCP_MAXSEG interval. > While we all like to avoid data copies (the whole reason for > RDDP to begin with), once we bite the bullet for software > CRCs (which is clearly the case with standard sockets), the > copy really isn't that painful. > > Regards, > Dwight _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 17:32:05 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28820 for ; Mon, 24 Feb 2003 17:32:05 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1OMen730841 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 17:40:49 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OMenp30838 for ; Mon, 24 Feb 2003 17:40:49 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28799 for ; Mon, 24 Feb 2003 17:31:34 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OMeCp30804; Mon, 24 Feb 2003 17:40:12 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1OMdrp30768 for ; Mon, 24 Feb 2003 17:39:53 -0500 Received: from mxic2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA28785 for ; Mon, 24 Feb 2003 17:30:38 -0500 (EST) From: Black_David@emc.com Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRHWB07>; Mon, 24 Feb 2003 17:34:29 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C933@corpmx14.us.dg.com> To: rddp@ietf.org Date: Mon, 24 Feb 2003 17:31:56 -0500 Importance: high X-Priority: 1 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] SF RDDP meeting time change Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , For those tracking the RDDP meeting times in San Francisco, the 2 hour meeting slot for RDDP just turned into a 2.5 hour slot and moved to Monday. This is not on the IETF web site yet, but will be soon, and is typical of the sort of agenda juggling that takes place before the IETF meeting times are finalized. I apologize if I haven't issued warnings that these meeting times do shift. Thanks, --David MONDAY, March 17, 2003 0900-1130 Morning Sessions APP apparea Applications Open Area Meeting GEN ipr Intellectual Property Rights WG INT dhc Dynamic Host Configuration WG SUB ccamp Common Control and Measurement Plane WG TSV nsis Next Steps in Signaling WG TSV rddp Remote Direct Data Placement WG TUESDAY, March 18, 2003 1700-1800 Afternoon Sessions IV APP lemonade License to Enhance Messaging Oriented Network Access for Diverse Endpoints PWG APP webdav WWW Distributed Authoring and Versioning WG INT itrace ICMP Traceback WG OPS grow Global Routing Operations BOF SEC ipseckey IPSEC KEYing information resource record WG TSV iptel IP Telephony WG TSV rddp Remote Direct Data Placement WG ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 18:13:17 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA29769 for ; Mon, 24 Feb 2003 18:13:16 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1ONM0i00658 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 18:22:00 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1ONLxp00655 for ; Mon, 24 Feb 2003 18:21:59 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA29762 for ; Mon, 24 Feb 2003 18:12:45 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1ONLHp00628; Mon, 24 Feb 2003 18:21:17 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1ONKUp00603 for ; Mon, 24 Feb 2003 18:20:30 -0500 Received: from ztxmail04.ztx.compaq.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA29707 for ; Mon, 24 Feb 2003 18:11:16 -0500 (EST) Received: from cceexg13.americas.cpqcorp.net (cceexg13.americas.cpqcorp.net [16.110.250.119]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id AADB64E36; Mon, 24 Feb 2003 17:15:07 -0600 (CST) Received: from cceexc16.americas.cpqcorp.net ([16.110.250.58]) by cceexg13.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Mon, 24 Feb 2003 17:15:07 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) Date: Mon, 24 Feb 2003 17:15:06 -0600 Message-ID: <01CA1B1F74A4EC40A9C887F7B58F8724031430E3@cceexc16.americas.cpqcorp.net> Thread-Topic: Optional CRCs ( was [rddp] Marker Interval ) Thread-Index: AcLcVGaeFa/l71HeTjmoJTAHGok0IAAAWVlQ From: "Barron, Dwight" To: "Williams, Jim" , X-OriginalArrivalTime: 24 Feb 2003 23:15:07.0259 (UTC) FILETIME=[91E3D4B0:01C2DC5A] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h1ONKUp00604 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit > -----Original Message----- > From: Williams, Jim [mailto:Jim.Williams@emulex.com] > Sent: Monday, February 24, 2003 4:23 PM > To: rddp@ietf.org > Subject: Optional CRCs ( was [rddp] Marker Interval ) > > > > > > -----Original Message----- > > From: Barron, Dwight [mailto:Dwight.Barron@hp.com] > > Sent: Monday, February 24, 2003 4:28 PM > > To: Somesh Gupta; rddp@ietf.org > > Subject: RE: [rddp] Marker Interval > > > > > > There are clearly a wide spectrum of implementation > > possibilities, but I like to focus on the endpoints first, > > then determine how many additional click-stops are necessary > > to adequately support the middle of the spectrum. Clearly, > > support of an ULP that wants to invoke RDDP over an > > unmodified TCP stack is an important goal, but optionality > > isn't free, so let's examine what we are really buying with > > negotiable marker intervals. I'll assume that the ULP using > > RDDP wants to see RDDP semantics (whatever that may be) and > > not interim RDDP headers and components. If you assume this > > RDDP "library" lives on top of unmodified sockets/TCP, then I > > believe there is no choice but for a receive stream to a) > > sequentially read every byte for purposes of CRC calculation > > and b) copy incoming stream data to separate the RDDP headers > > from ULP payload. This is especially true of tagged messages > > where the ULP is using a contiguous memory address range to > > reference data. On the transmit side, the R! > > DDP library will still have to touch every byte for CRC > > calculation purposes, > > I think if we are serious about supporting unmodified TCP, then > the CRC should be optional. NFS typically runs with only TCP/UDP > I think there is a significant delta between "serious about supporting" and "serious about optimizing for". Quoting from the RDDP charter (http://www.ietf.org/html.charters/rddp-charter.html) "RDDP capabilities refer to the ability to place data directly from the NIC into application buffers, without intensive CPU usage. This strategy avoids the costs of data copying and enables using IP as a method for high speed buffer to buffer transfer, allowing IP to replace special purpose networks currently in use." I believe we know how to optimize to deliver this behavior at one end of the implementation spectrum, but have no illusions of delivering this behavior with a pure software implementation. > checksum for protection. Not a good idea to do this over the > public internet, but then critical data traveling through > the public internet will (or should) use IPSec protection. > And where does the IPSEC processing power come from? I would be willing to bet there is a large overlap of candidate environments that need to run RDDP in unmodified software stacks that will need to run IPSEC in that same unmodified software stack. IPSEC is great in its own right, but is certainly not a good tradeoff when a CRC would suffice. > iSCSI is a precedent for optional CRCs. Data errors in iSCSI > or NFS are equally catastrophic as RDDP. Tradeoff can be > made by user as to required protection, quality and reliability > of network infrastructure, etc., to decide whether to use CRCs. > > > but may try to avoid copying by using > > multiple send calls. But at what point does copying become > > cheaper than multiple send calls? Especially if the sender is > > at least *attempting* to provide header alignment, which > > would mean adding RDDP headers for each TCP_MAXSEG interval. > > While we all like to avoid data copies (the whole reason for > > RDDP to begin with), once we bite the bullet for software > > CRCs (which is clearly the case with standard sockets), the > > copy really isn't that painful. > > > > Regards, > > Dwight > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 18:24:03 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA29938 for ; Mon, 24 Feb 2003 18:24:02 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1ONWkU00998 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 18:32:46 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1ONWkp00995 for ; Mon, 24 Feb 2003 18:32:46 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA29914 for ; Mon, 24 Feb 2003 18:23:31 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1ONW1p00982; Mon, 24 Feb 2003 18:32:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1ONVep00963 for ; Mon, 24 Feb 2003 18:31:40 -0500 Received: from ztxmail04.ztx.compaq.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA29901 for ; Mon, 24 Feb 2003 18:22:25 -0500 (EST) Received: from cceexg12.americas.cpqcorp.net (cceexg12.americas.cpqcorp.net [16.110.250.124]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id 261CF4DA1 for ; Mon, 24 Feb 2003 17:26:18 -0600 (CST) Received: from cceexc18.americas.cpqcorp.net ([16.110.250.64]) by cceexg12.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Mon, 24 Feb 2003 17:26:17 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----_=_NextPart_001_01C2DC5C.20BB4D9E" Date: Mon, 24 Feb 2003 17:26:16 -0600 Message-ID: <4D027986353D1341ADA4F2A4F8A170762975A670@cceexc18.americas.cpqcorp.net> Thread-Topic: I-D ACTION:draft-culley-iwarp-mpa-02.txt Thread-Index: AcLcXB9ERwPQ3HohTw28JyDYDU4uLA== From: "Culley, Paul" To: X-OriginalArrivalTime: 24 Feb 2003 23:26:17.0950 (UTC) FILETIME=[21A727E0:01C2DC5C] Subject: [rddp] I-D ACTION:draft-culley-iwarp-mpa-02.txt Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , This is a multi-part message in MIME format. ------_=_NextPart_001_01C2DC5C.20BB4D9E Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable I'm not sure if anyone has seen this yet, so... A New Internet-Draft is available from the on-line Internet-Drafts directories. Title : Marker PDU Aligned Framing for TCP Specification Author(s) : P. Culley et al. Filename : draft-culley-iwarp-mpa-02.txt,.pdf Pages : 35 Date : 2003-2-20 =09 A framing protocol is defined for TCP that is fully compliant with applicable TCP RFCs and fully interoperable with existing TCP implementations. The framing mechanism is designed to work as an 'adaptation layer' between TCP and the Direct Data Placement [DDP] protocol, preserving the reliable, in-order delivery of TCP, while adding the preservation of higher-level protocol record boundaries that DDP requires. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-culley-iwarp-mpa-02.txt To remove yourself from the IETF Announcement list, send a message to=20 ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-culley-iwarp-mpa-02.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html=20 or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-culley-iwarp-mpa-02.txt". =09 NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. =09 =09 Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. =20 Paul R. Culley HP Senior Fellow 281-514-5543 =20 ------_=_NextPart_001_01C2DC5C.20BB4D9E Content-Type: text/html; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Message 
I'm not sure if anyone has =
seen this yet, so...
A New Internet-Draft is available =
from the on-line Internet-Drafts directories.


	Title		: Marker PDU Aligned Framing for TCP Specification
	Author(s)	: P. Culley et al.
	Filename	: draft-culley-iwarp-mpa-02.txt,.pdf
	Pages		: 35
	Date		: 2003-2-20
=09
A framing protocol is defined for TCP that is fully compliant with
applicable TCP RFCs and fully interoperable with existing TCP
implementations. The framing mechanism is designed to work as an
'adaptation layer' between TCP and the Direct Data Placement [DDP]
protocol, preserving the reliable, in-order delivery of TCP, while
adding the preservation of higher-level protocol record boundaries
that DDP requires.

A URL for this Internet-Draft is:
http://www.ietf.org/internet-drafts/draft-culley-iwarp-mpa-02.txt

To remove yourself from the IETF Announcement list, send a message to=20
ietf-announce-request with the word unsubscribe in the body of the =
message.

Internet-Drafts are also available by anonymous FTP. Login with the =
username
"anonymous" and a password of your e-mail address. After logging in,
type "cd internet-drafts" and then
	"get draft-culley-iwarp-mpa-02.txt".

A list of Internet-Drafts directories can be found in
http://www.ietf.org/shadow.html<=
/A>=20
or ftp://ftp.ietf.org/iet=
f/1shadow-sites.txt


Internet-Drafts can also be obtained by e-mail.

Send a message to:
	mailserv@ietf.org.
In the body type:
	"FILE /internet-drafts/draft-culley-iwarp-mpa-02.txt".
=09
NOTE:	The mail server at ietf.org can return the document in
	MIME-encoded form by using the "mpack" utility.  To use this
	feature, insert the command "ENCODING mime" before the "FILE"
	command.  To decode the response(s), you will need "munpack" or
	a MIME-compliant mail reader.  Different MIME-compliant mail readers
	exhibit different behavior, especially when dealing with
	"multipart" MIME messages (i.e. documents which have been split
	up into multiple messages), so check your local documentation on
	how to manipulate these messages.
	=09
	=09
Below is the data which will enable a MIME compliant mail reader
implementation to automatically retrieve the ASCII version of the
Internet-Draft.
<ftp://ftp.ietf.org/internet-drafts/draft-culley-iwarp-mpa-02.txt>=
 
Paul R. = Culley
HP Senior = Fellow
281-514-5543
 
------_=_NextPart_001_01C2DC5C.20BB4D9E-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 18:47:09 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA00572 for ; Mon, 24 Feb 2003 18:47:09 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1ONtr902524 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 18:55:53 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1ONtrp02521 for ; Mon, 24 Feb 2003 18:55:53 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA00547 for ; Mon, 24 Feb 2003 18:46:37 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1ONt8p02501; Mon, 24 Feb 2003 18:55:08 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1ONsjp02475 for ; Mon, 24 Feb 2003 18:54:45 -0500 Received: from palrel11.hp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA00518 for ; Mon, 24 Feb 2003 18:45:29 -0500 (EST) Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel11.hp.com (Postfix) with ESMTP id EB8B31C00EF7 for ; Mon, 24 Feb 2003 15:49:22 -0800 (PST) Received: from mk731915.cup.hp.com ([15.113.161.2]) by esmail.cup.hp.com (8.9.3/8.8.6) with ESMTP id PAA03894 for ; Mon, 24 Feb 2003 15:46:37 -0800 (PST) Message-Id: <5.1.0.14.2.20030224152530.03255b40@esmail.cup.hp.com> X-Sender: krause@esmail.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Mon, 24 Feb 2003 15:30:51 -0800 To: rddp@ietf.org From: Michael Krause Subject: Re: Optional CRCs ( was [rddp] Marker Interval ) In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B9000@xbl.ma.emulex.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 05:23 PM 2/24/2003 -0500, Williams, Jim wrote: > > -----Original Message----- > > From: Barron, Dwight [mailto:Dwight.Barron@hp.com] > > Sent: Monday, February 24, 2003 4:28 PM > > To: Somesh Gupta; rddp@ietf.org > > Subject: RE: [rddp] Marker Interval > > > > > > There are clearly a wide spectrum of implementation > > possibilities, but I like to focus on the endpoints first, > > then determine how many additional click-stops are necessary > > to adequately support the middle of the spectrum. Clearly, > > support of an ULP that wants to invoke RDDP over an > > unmodified TCP stack is an important goal, but optionality > > isn't free, so let's examine what we are really buying with > > negotiable marker intervals. I'll assume that the ULP using > > RDDP wants to see RDDP semantics (whatever that may be) and > > not interim RDDP headers and components. If you assume this > > RDDP "library" lives on top of unmodified sockets/TCP, then I > > believe there is no choice but for a receive stream to a) > > sequentially read every byte for purposes of CRC calculation > > and b) copy incoming stream data to separate the RDDP headers > > from ULP payload. This is especially true of tagged messages > > where the ULP is using a contiguous memory address range to > > reference data. On the transmit side, the R! > > DDP library will still have to touch every byte for CRC > > calculation purposes, > >I think if we are serious about supporting unmodified TCP, then >the CRC should be optional. NFS typically runs with only TCP/UDP >checksum for protection. Not a good idea to do this over the >public internet, but then critical data traveling through >the public internet will (or should) use IPSec protection. Isn't SCTP CRC protection required per chunk? The reason to have a CRC for MPA is to provide data integrity below DDP/RDMA / ULP which has been demonstrated as a concern during the development of iSCSI. Do we really need to go back through the entire IPS debate as to why CRCs are good things here or accept that a CRC at or above the transport but below DDP/RDMA / ULP is a good thing? Mike >iSCSI is a precedent for optional CRCs. Data errors in iSCSI >or NFS are equally catastrophic as RDDP. Tradeoff can be >made by user as to required protection, quality and reliability >of network infrastructure, etc., to decide whether to use CRCs. > > > but may try to avoid copying by using > > multiple send calls. But at what point does copying become > > cheaper than multiple send calls? Especially if the sender is > > at least *attempting* to provide header alignment, which > > would mean adding RDDP headers for each TCP_MAXSEG interval. > > While we all like to avoid data copies (the whole reason for > > RDDP to begin with), once we bite the bullet for software > > CRCs (which is clearly the case with standard sockets), the > > copy really isn't that painful. > > > > Regards, > > Dwight >_______________________________________________ >rddp mailing list >rddp@ietf.org >https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 19:12:02 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01162 for ; Mon, 24 Feb 2003 19:12:02 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1P0Kl303993 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 19:20:47 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1P0Kkp03990 for ; Mon, 24 Feb 2003 19:20:46 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01156 for ; Mon, 24 Feb 2003 19:11:31 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1P0K2p03964; Mon, 24 Feb 2003 19:20:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1P0JFp03925 for ; Mon, 24 Feb 2003 19:19:15 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01114 for ; Mon, 24 Feb 2003 19:10:00 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 695D3245079; Mon, 24 Feb 2003 19:13:52 -0500 (EST) Date: Mon, 24 Feb 2003 18:15:19 -0600 From: Caitlin Bestler Subject: Re: Optional CRCs ( was [rddp] Marker Interval ) To: Michael Krause Cc: rddp@ietf.org X-Priority: 3 In-Reply-To: <5.1.0.14.2.20030224152530.03255b40@esmail.cup.hp.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/24/03, Michael Krause wrote: >Isn't SCTP CRC protection required per chunk? The reason >to have a CRC for MPA is to provide data integrity below >DDP/RDMA / ULP which has been demonstrated as a concern >during the development of iSCSI. Do we really need to go >back through the entire IPS debate as to why CRCs are good >things here or accept that a CRC at or above the transport >but below DDP/RDMA / ULP is a good thing? SCTP provides CRC-32c on the SCTP Packet, not individual Data Chunks. The SCTP Common Header is defined on page 17: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Source Port Number | Destination Port Number | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Verification Tag | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Checksum | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ Data Chunks are defined on page 22: 0 1 2 3 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Type = 0 | Reserved|U|B|E| Length | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | TSN | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Stream Identifier S | Stream Sequence Number n | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ | Payload Protocol Identifier | +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ \ \ / User Data (seq n of Stream S) / \ \ +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ My understanding was that the need for a CRC above TCP had more to do with its size (16 bits) and algorithm (less sensitive than a full CRC) than with the layer. Additionally, because SCTP performs end-to-end PMTU discovery there would be no reason for a middle box to rebundle SCTP chunks in flight. Indeed on page 83, RFC2960 specifies that once a user message is fragmented into Data Chunks it cannot be refragmented. While bundling is a different process, the spirit of relying upon PMTU discovery should apply. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Mon Feb 24 19:45:03 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01644 for ; Mon, 24 Feb 2003 19:45:03 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1P0rm905549 for rddp-archive@odin.ietf.org; Mon, 24 Feb 2003 19:53:48 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1P0rmp05546 for ; Mon, 24 Feb 2003 19:53:48 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01639 for ; Mon, 24 Feb 2003 19:44:32 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1P0r3p05531; Mon, 24 Feb 2003 19:53:03 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1P0qrp05517 for ; Mon, 24 Feb 2003 19:52:53 -0500 Received: from dmz1.silverbacksystems.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA01634 for ; Mon, 24 Feb 2003 19:43:36 -0500 (EST) Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id 3BDC5D145; Mon, 24 Feb 2003 16:47:17 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id ED15F199A3; Mon, 24 Feb 2003 16:47:16 -0800 (PST) Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="us-ascii" Date: Mon, 24 Feb 2003 16:46:45 -0800 From: "Somesh Gupta" Importance: Normal In-Reply-To: <5.1.0.14.2.20030224113354.0200fed8@esmail.cup.hp.com> Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 13495-3CB29BED; Mon, 24 Feb 2003 16:46:33 -0800 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id 90DC4199CC; Mon, 24 Feb 2003 16:46:33 -0800 (PST) Subject: RE: [rddp] Marker Interval To: "Michael Krause" , X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Software implementation should not be considered the same as user space apps. Many of the iSCSI implementations are implemented in the kernel and so is NFS. I assume that if one of main apps for RDMA is storage, then kernel based implementations (iSCSI and some NFS variant) will be common. In such as case there is no data copy needed and other costs will continue to dominate. Regarding the CRC debate, I don't believe that NAS or iSCSI is broken because CRC is not part of the spec or not implemented by software based iSCSI stacks. It is very useful to look to take a deeper look at the paper behind this whole debate. Most of the bugs were traced to bad implementations at the end systems. A NIC that corrupts data before performing checksum calculations will corrupt data before CRC calculation. The CRC will provide additional protection as the data passes through switches but there are no studies focused on those errors and their escape rates from TCP checksum. Somesh > -----Original Message----- > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > Michael Krause > Sent: Monday, February 24, 2003 12:08 PM > To: rddp@ietf.org > Subject: RE: [rddp] Marker Interval > > > At 11:10 AM 2/24/2003 -0800, Somesh Gupta wrote: > > > > > -----Original Message----- > > > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > > > Michael Krause > > > Sent: Monday, February 24, 2003 10:04 AM > > > To: rddp@ietf.org > > > Subject: RE: [rddp] Marker Interval > > > > > > > > > At 09:42 AM 2/24/2003 -0800, Somesh Gupta wrote: > > > >The discussion is related to existing software stacks and > > > >interfaces. For storage protocols (as is for iSCSI), this > > > >forms the first wave of client side implementations. > > > > > > Again this is a function of implementation and a well-layered > > > implementation can be delivered without requiring any network stack > > > modifications if desired. Same applies to software based > iSCSI which has > > > been demonstrated by multiple vendors without network stack > > > modifications. Some of these implementations now deliver sufficient > > > performance that current customers using them are not raising > any major > > > objections. Given this can be done with iSCSI and markers are > > > clearly less > > > complex to implement / support in software than iSCSI, one might > > > argue that > > > the complexity / cost issues are not as high as some assert. > > > > I don't know what the point here. > > - That markers are not complex to implement nor are they that major of a > performance loss as the data is already being moved from one buffer to > another since the hardware could not perform the direct placement when > using a software stack. > > - That markers can be implemented on top of an existing network stack > keeping things well layered which is one of the objectives being > stated on > this e-mail string. > > >You can run a distributed database over a software > implementation which is > >clearly less complex > > than markers. How does that imply that the cost of markers is > not high? > > I've requested a definition of "high" as the discussion to date > is perhaps > too subjective. I described a potential implementation to > illustrate what > can be done that IMO is not complex or a major performance inhibitor. > > >In terms of complexity, it is close to 0 complexity to implement markers > >in software (especially when you don't have to modify the stack > to do so). > > Then we are in agreement that the complexity is not high in > software for a > well-layered implementation. Is there any remaining disagreement in this > regard? > > Mike > > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 25 06:50:41 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA25315 for ; Tue, 25 Feb 2003 06:50:41 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1PBxe321595 for rddp-archive@odin.ietf.org; Tue, 25 Feb 2003 06:59:40 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PBxep21592 for ; Tue, 25 Feb 2003 06:59:40 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA25252 for ; Tue, 25 Feb 2003 06:50:10 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PBwpp21495; Tue, 25 Feb 2003 06:58:51 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PBtvp21278 for ; Tue, 25 Feb 2003 06:55:57 -0500 Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id GAA25038; Tue, 25 Feb 2003 06:46:28 -0500 (EST) Message-Id: <200302251146.GAA25038@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: rddp@ietf.org From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Date: Tue, 25 Feb 2003 06:46:27 -0500 Subject: [rddp] I-D ACTION:draft-ietf-rddp-ddp-00.txt Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Remote Direct Data Placement Working Group of the IETF. Title : Direct Data Placement over Reliable Transports Author(s) : H. Shah et al. Filename : draft-ietf-rddp-ddp-00.txt Pages : 39 Date : 2003-2-24 The Direct Data Placement protocol provides information to Place the incoming data directly into an upper layer protocol's receive buffer without intermediate buffers. This removes excess CPU and memory utilization associated with transferring data through the intermediate buffers. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-ietf-rddp-ddp-00.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-ietf-rddp-ddp-00.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-ietf-rddp-ddp-00.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <2003-2-24142556.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-ietf-rddp-ddp-00.txt --OtherAccess Content-Type: Message/External-body; name="draft-ietf-rddp-ddp-00.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <2003-2-24142556.I-D@ietf.org> --OtherAccess-- --NextPart-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 25 09:26:52 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA01875 for ; Tue, 25 Feb 2003 09:26:52 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1PEZsG00504 for rddp-archive@odin.ietf.org; Tue, 25 Feb 2003 09:35:54 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PEZsp00501 for ; Tue, 25 Feb 2003 09:35:54 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA01869 for ; Tue, 25 Feb 2003 09:26:21 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PEZEp00474; Tue, 25 Feb 2003 09:35:14 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PEYrp00438 for ; Tue, 25 Feb 2003 09:34:53 -0500 Received: from mxic1.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA01854 for ; Tue, 25 Feb 2003 09:25:19 -0500 (EST) From: Black_David@emc.com Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1KDZ14GL>; Tue, 25 Feb 2003 09:27:47 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C939@corpmx14.us.dg.com> To: krause@cup.hp.com, rddp@ietf.org Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) Date: Tue, 25 Feb 2003 09:25:52 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > Isn't SCTP CRC protection required per chunk? Actually, it's per packet (in the SCTP common header), as Caitlin has pointed out. SCTP did not have a checksum that needed to be preserved, in contrast to TCP. The IPS WG considered the TCP alternate checksum mechanism in RFC 1146 as a means of using a CRC to replace the TCP checksum, but that mechanism is undeployable in practice due to what it breaks, even though it would have been a cleaner solution in principle. > The reason to have a CRC for > MPA is to provide data integrity below DDP/RDMA / ULP which has been > demonstrated as a concern during the development of iSCSI. Do we really > need to go back through the entire IPS debate as to why CRCs are good > things here or accept that a CRC at or above the transport but below > DDP/RDMA / ULP is a good thing? We don't need to go back through that debate, because the question isn't about "good to implement" - it's about "mandatory to use", which is a very different issue. CRCs are optional for iSCSI, and NFS (another important client for RDDP) has never needed CRCs, as the TCP checksum has been adequate for its purposes (yes, a CRC would be an improvement). Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 25 09:39:39 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA02094 for ; Tue, 25 Feb 2003 09:39:39 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1PEmgc01720 for rddp-archive@odin.ietf.org; Tue, 25 Feb 2003 09:48:42 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PEmgp01717 for ; Tue, 25 Feb 2003 09:48:42 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA02089 for ; Tue, 25 Feb 2003 09:39:07 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PEm2p01686; Tue, 25 Feb 2003 09:48:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PElgp01669 for ; Tue, 25 Feb 2003 09:47:42 -0500 Received: from palrel11.hp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA02083 for ; Tue, 25 Feb 2003 09:38:07 -0500 (EST) Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel11.hp.com (Postfix) with ESMTP id ECF3B1C0165F for ; Tue, 25 Feb 2003 06:42:01 -0800 (PST) Received: from mk731915.cup.hp.com ([15.113.161.2]) by esmail.cup.hp.com (8.9.3/8.8.6) with ESMTP id GAA17466 for ; Tue, 25 Feb 2003 06:39:19 -0800 (PST) Message-Id: <5.1.0.14.2.20030225063339.01ed9be0@esmail.cup.hp.com> X-Sender: krause@esmail.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Tue, 25 Feb 2003 06:38:52 -0800 To: From: Michael Krause Subject: RE: [rddp] Marker Interval In-Reply-To: References: <5.1.0.14.2.20030224113354.0200fed8@esmail.cup.hp.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 04:46 PM 2/24/2003 -0800, Somesh Gupta wrote: >Software implementation should not be considered the same as >user space apps. > >Many of the iSCSI implementations are implemented in the kernel and >so is NFS. I assume that if one of main apps for RDMA is storage, >then kernel based implementations (iSCSI and some NFS variant) will >be common. In such as case there is no data copy needed and other >costs will continue to dominate. There may not be a data copy on outbound but certainly there will be on in-bound for many implementations of iSCSI, NFS, etc. as the operations do not necessarily align with the target buffers. >Regarding the CRC debate, I don't believe that NAS or iSCSI is >broken because CRC is not part of the spec or not implemented >by software based iSCSI stacks. No one stated they are broken nor that neither of these types of solutions do not use CRC in their software based stacks. Same argument applies to solutions implemented on top of SCTP which provides a CRC I believe in place of the Addler checksum (need to go back and re-read that RFC again to verify). >It is very useful to look to take a deeper look at the paper >behind this whole debate. Most of the bugs were traced to bad >implementations at the end systems. A NIC that corrupts data >before performing checksum calculations will corrupt data >before CRC calculation. CRCs also exist to mitigate unprotected intermediated devices which may not provide ECC protection of their associated memories. As hardware moves to ever smaller processes, end-to-end protection requirements increase. I've heard many a customer state that they will not accept technology that does not provide end-to-end protection as silent data corruption is not acceptable - it does not matter whether it is in someone's implementation as they are examining the solution as a whole and don't want to have to take into account every physical device along the way. >The CRC will provide additional protection as the data passes >through switches but there are no studies focused on those >errors and their escape rates from TCP checksum. > >Somesh > > > -----Original Message----- > > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > > Michael Krause > > Sent: Monday, February 24, 2003 12:08 PM > > To: rddp@ietf.org > > Subject: RE: [rddp] Marker Interval > > > > > > At 11:10 AM 2/24/2003 -0800, Somesh Gupta wrote: > > > > > > > > -----Original Message----- > > > > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > > > > Michael Krause > > > > Sent: Monday, February 24, 2003 10:04 AM > > > > To: rddp@ietf.org > > > > Subject: RE: [rddp] Marker Interval > > > > > > > > > > > > At 09:42 AM 2/24/2003 -0800, Somesh Gupta wrote: > > > > >The discussion is related to existing software stacks and > > > > >interfaces. For storage protocols (as is for iSCSI), this > > > > >forms the first wave of client side implementations. > > > > > > > > Again this is a function of implementation and a well-layered > > > > implementation can be delivered without requiring any network stack > > > > modifications if desired. Same applies to software based > > iSCSI which has > > > > been demonstrated by multiple vendors without network stack > > > > modifications. Some of these implementations now deliver sufficient > > > > performance that current customers using them are not raising > > any major > > > > objections. Given this can be done with iSCSI and markers are > > > > clearly less > > > > complex to implement / support in software than iSCSI, one might > > > > argue that > > > > the complexity / cost issues are not as high as some assert. > > > > > > I don't know what the point here. > > > > - That markers are not complex to implement nor are they that major of a > > performance loss as the data is already being moved from one buffer to > > another since the hardware could not perform the direct placement when > > using a software stack. > > > > - That markers can be implemented on top of an existing network stack > > keeping things well layered which is one of the objectives being > > stated on > > this e-mail string. > > > > >You can run a distributed database over a software > > implementation which is > > >clearly less complex > > > than markers. How does that imply that the cost of markers is > > not high? > > > > I've requested a definition of "high" as the discussion to date > > is perhaps > > too subjective. I described a potential implementation to > > illustrate what > > can be done that IMO is not complex or a major performance inhibitor. > > > > >In terms of complexity, it is close to 0 complexity to implement markers > > >in software (especially when you don't have to modify the stack > > to do so). > > > > Then we are in agreement that the complexity is not high in > > software for a > > well-layered implementation. Is there any remaining disagreement in this > > regard? > > > > Mike > > > > > > _______________________________________________ > > rddp mailing list > > rddp@ietf.org > > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 25 09:44:37 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA02187 for ; Tue, 25 Feb 2003 09:44:36 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1PErdC01953 for rddp-archive@odin.ietf.org; Tue, 25 Feb 2003 09:53:39 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PErdp01950 for ; Tue, 25 Feb 2003 09:53:39 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA02169 for ; Tue, 25 Feb 2003 09:44:05 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PEr1p01917; Tue, 25 Feb 2003 09:53:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PEq7p01871 for ; Tue, 25 Feb 2003 09:52:07 -0500 Received: from mxic2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA02148 for ; Tue, 25 Feb 2003 09:42:33 -0500 (EST) From: Black_David@emc.com Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRHW0M6>; Tue, 25 Feb 2003 09:46:16 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C93A@corpmx14.us.dg.com> To: Dwight.Barron@hp.com, rddp@ietf.org Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) Date: Tue, 25 Feb 2003 09:43:41 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Dwight, > > I think if we are serious about supporting unmodified TCP, then > > the CRC should be optional. NFS typically runs with only TCP/UDP > > > I think there is a significant delta between "serious about > supporting" and "serious about optimizing for". An alternate viewpoint is that "serious about supporting" includes "not placing unreasonable implementation requirements upon". NFS as currently used does not need CRCs (although NFS did learn the hard way that turning on checksums was a real good idea), and ... > > checksum for protection. Not a good idea to do this over the > > public internet, but then critical data traveling through > > the public internet will (or should) use IPSec protection. > > > And where does the IPSEC processing power come from? I would be > willing to bet there is a large overlap of candidate environments > that need to run RDDP in unmodified software stacks that will > need to run IPSEC in that same unmodified software stack. IPSEC > is great in its own right, but is certainly not a good tradeoff > when a CRC would suffice. To begin with, there's quite a bit of OSV support for IPsec in existing stacks today. The point here is not that IPsec is computationally cheap (it's not), but rather that in an environment where IPsec is required for security policy reasons and the IPsec coverage is similar to the CRC coverage, the CRC may not be needed as the 96-bit IPsec MAC provides a considerably stronger integrity check - see Section 3.6 of draft-ietf-ips-security-19.txt. As Jim Williams wrote: > > iSCSI is a precedent for optional CRCs. And there are two scenarios discussed above where the CRC does not appear to be required. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 25 10:26:39 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA03111 for ; Tue, 25 Feb 2003 10:26:39 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1PFZhO04608 for rddp-archive@odin.ietf.org; Tue, 25 Feb 2003 10:35:43 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PFZhp04605 for ; Tue, 25 Feb 2003 10:35:43 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA03076 for ; Tue, 25 Feb 2003 10:26:08 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PFZ2p04564; Tue, 25 Feb 2003 10:35:02 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PFQZp04266 for ; Tue, 25 Feb 2003 10:26:35 -0500 Received: from pheriche.sun.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA02929 for ; Tue, 25 Feb 2003 10:17:00 -0500 (EST) Received: from phys-hanwk16-1.ebay.sun.com ([129.149.1.5]) by pheriche.sun.com (8.9.3+Sun/8.9.3) with ESMTP id IAA07118 for ; Tue, 25 Feb 2003 08:20:54 -0700 (MST) Received: from Sun.COM (vpn-129-147-154-54.Central.Sun.COM [129.147.154.54]) by phys-hanwk16-1.ebay.sun.com (8.11.6+Sun/8.10.2/ENSMAIL,v2.1p1) with ESMTP id h1PFKrL23958 for ; Tue, 25 Feb 2003 07:20:53 -0800 (PST) Message-ID: <3E5B8A68.2020301@Sun.COM> Date: Tue, 25 Feb 2003 09:23:20 -0600 From: David Robinson User-Agent: Mozilla/5.0 (Windows; U; Win98; en-US; rv:1.0.2) Gecko/20021120 Netscape/7.01 X-Accept-Language: en-us, en MIME-Version: 1.0 To: rddp@ietf.org Subject: Re: Optional CRCs ( was [rddp] Marker Interval ) References: <277DD60FB639D511AC0400B0D068B71E0564C93A@corpmx14.us.dg.com> Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Black_David@emc.com wrote: >>>I think if we are serious about supporting unmodified TCP, then >>>the CRC should be optional. NFS typically runs with only TCP/UDP >>I think there is a significant delta between "serious about >>supporting" and "serious about optimizing for". > An alternate viewpoint is that "serious about supporting" includes > "not placing unreasonable implementation requirements upon". NFS > as currently used does not need CRCs (although NFS did learn the > hard way that turning on checksums was a real good idea), and ... Just so no one is confused, while NFS typically runs over TCP or UDP using their default checksums and most find that an adequate solution, NFS also can optionally use RPCSEC_GSS to add integrity checking if desired. So NFS has belts and suspenders, although most just wear a belt. Adding yet another integrity check is not required for NFS. -David _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 25 10:55:46 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA04335 for ; Tue, 25 Feb 2003 10:55:46 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1PG4p607471 for rddp-archive@odin.ietf.org; Tue, 25 Feb 2003 11:04:51 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PG4pp07468 for ; Tue, 25 Feb 2003 11:04:51 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA04320 for ; Tue, 25 Feb 2003 10:55:15 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PG47p07398; Tue, 25 Feb 2003 11:04:07 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PG3Hp07352 for ; Tue, 25 Feb 2003 11:03:17 -0500 Received: from ztxmail05.ztx.compaq.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA04224 for ; Tue, 25 Feb 2003 10:53:41 -0500 (EST) Received: from cceexg13.americas.cpqcorp.net (cceexg13.americas.cpqcorp.net [16.110.250.119]) by ztxmail05.ztx.compaq.com (Postfix) with ESMTP id 72E78398C; Tue, 25 Feb 2003 09:57:35 -0600 (CST) Received: from cceexc16.americas.cpqcorp.net ([16.110.250.58]) by cceexg13.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Tue, 25 Feb 2003 09:57:35 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) Date: Tue, 25 Feb 2003 09:57:34 -0600 Message-ID: <01CA1B1F74A4EC40A9C887F7B58F8724031430E6@cceexc16.americas.cpqcorp.net> Thread-Topic: Optional CRCs ( was [rddp] Marker Interval ) Thread-Index: AcLc3LKapeF4mkS1QmO5ebdHHA8HQwABRClA From: "Barron, Dwight" To: , X-OriginalArrivalTime: 25 Feb 2003 15:57:35.0114 (UTC) FILETIME=[9CCE26A0:01C2DCE6] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h1PG3Hp07353 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Tuesday, February 25, 2003 8:44 AM > To: Barron, Dwight; rddp@ietf.org > Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) > > > Dwight, > > > > I think if we are serious about supporting unmodified TCP, then > > > the CRC should be optional. NFS typically runs with only TCP/UDP > > > > > I think there is a significant delta between "serious about > > supporting" and "serious about optimizing for". > > An alternate viewpoint is that "serious about supporting" includes > "not placing unreasonable implementation requirements upon". But unreasonable from which perspective. If we were assuming symmetric implementations at each end, with roughly equal costs for performing a function, then optionality can be evaluated from end user trade-offs, usually functionality (like CRC) vs. performance or cost to implement or whatever. Asymmetric implementations mean that one end will (willingly) bear more of a burden for the benefit of the other. I have not heard nor seen any evidence that a CRC is an unreasonable burden to implement in hardware. It is standard practice for modern IETF protocols (SCTP), other modern storage and datacenter interconnect technologies (FC, IB). Why do we want to burden over-burdened administrators with yet another option (would you like data integrity with that?). Note that I am not trying to be unsympathetic to the needs and desires of efficient software implementations, I am trying to balance the realities asymmetric implementations and the need for optionality, both from an implement! ation/test perspective and the end user administrator. > NFS > as currently used does not need CRCs (although NFS did learn the > hard way that turning on checksums was a real good idea), and ... > > > > checksum for protection. Not a good idea to do this over the > > > public internet, but then critical data traveling through > > > the public internet will (or should) use IPSec protection. > > > > > And where does the IPSEC processing power come from? I would be > > willing to bet there is a large overlap of candidate environments > > that need to run RDDP in unmodified software stacks that will > > need to run IPSEC in that same unmodified software stack. IPSEC > > is great in its own right, but is certainly not a good tradeoff > > when a CRC would suffice. > > To begin with, there's quite a bit of OSV support for IPsec in > existing stacks today. The point here is not that IPsec is > computationally cheap (it's not), but rather that in an environment > where IPsec is required for security policy reasons and the IPsec > coverage is similar to the CRC coverage, the CRC may not be needed > as the 96-bit IPsec MAC provides a considerably stronger integrity > check - see Section 3.6 of draft-ietf-ips-security-19.txt. As The pervasive OSV support was exactly my point, unmodified existing software stacks. However, I have been told, but have not personally confirmed, that there is no easy/standard way for a ULP above TCP to discover if IPSEC is actually being invoked on a given connection with these existing stacks. If this is actually the case, then exploiting overlapping IPSEC/CRC coverage would seem to be left as yet another exercise for the administrator. > Jim Williams wrote: > > > > iSCSI is a precedent for optional CRCs. > > And there are two scenarios discussed above where the CRC does not > appear to be required. Again, these were/are very valid architectural decisions when software implementations were heavily weighted in the decision process. Hardware equivalent architectural decisions (e.g. FC) that carry the exact same class of storage traffic always have CRCs enabled. > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > black_david@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 25 11:00:34 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA04587 for ; Tue, 25 Feb 2003 11:00:34 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1PG9de08582 for rddp-archive@odin.ietf.org; Tue, 25 Feb 2003 11:09:39 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PG9dp08579 for ; Tue, 25 Feb 2003 11:09:39 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA04566 for ; Tue, 25 Feb 2003 11:00:03 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PG91p08555; Tue, 25 Feb 2003 11:09:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PG8op08536 for ; Tue, 25 Feb 2003 11:08:50 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA04536 for ; Tue, 25 Feb 2003 10:59:14 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id C4D4D24525D; Tue, 25 Feb 2003 11:03:08 -0500 (EST) Date: Tue, 25 Feb 2003 10:04:37 -0600 From: Caitlin Bestler Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) To: Black_David@emc.com, rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C939@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/25/03, Black_David@emc.com wrote: >We don't need to go back through that debate, because the >question isn't about "good to implement" - it's about >"mandatory to use", which is a very different issue. CRCs >are optional for iSCSI, and NFS (another important client >for RDDP) has never needed CRCs, as the TCP checksum has >been adequate for its purposes (yes, a CRC would be an >improvement). Purely subjective observation here, but it struck me that the strongest champions of RDDP having at *least* a 32 bit CRC were those with heavy NFS experience. Perhaps the feeling is that whether or not NFS *required* a CRC, it truly always needed it. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 25 11:08:45 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA04744 for ; Tue, 25 Feb 2003 11:08:44 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1PGHou08926 for rddp-archive@odin.ietf.org; Tue, 25 Feb 2003 11:17:50 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PGHnp08923 for ; Tue, 25 Feb 2003 11:17:49 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA04733 for ; Tue, 25 Feb 2003 11:08:12 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PGH4p08898; Tue, 25 Feb 2003 11:17:04 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PGGTp08864 for ; Tue, 25 Feb 2003 11:16:29 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA04711 for ; Tue, 25 Feb 2003 11:06:53 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 5584924521E; Tue, 25 Feb 2003 11:10:47 -0500 (EST) Date: Tue, 25 Feb 2003 10:12:15 -0600 From: Caitlin Bestler Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) To: Black_David@emc.com, rddp@ietf.org X-Priority: 3 In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C93A@corpmx14.us.dg.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/25/03, Black_David@emc.com wrote: >An alternate viewpoint is that "serious about supporting" >includes "not placing unreasonable implementation >requirements upon". NFS as currently used does not need >CRCs (although NFS did learn the hard way that turning on >checksums was a real good idea), and ... One aspect of avoiding unreasonable implementation requirements that I believe we have not paid enough attention to is trying to increase reusability of software modules (and gates for hardware solutions). Specifically, would it be a valid goal to make DDP's adaptation to TCP be compatible with iSCSI's? Compatible would not require them to be compatible. For example, RDDP could require feature that iSCSI considered to be optional. The benefits in logic sharing would still apply. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 25 11:39:46 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA05770 for ; Tue, 25 Feb 2003 11:39:46 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1PGmqj11067 for rddp-archive@odin.ietf.org; Tue, 25 Feb 2003 11:48:52 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PGmqp11064 for ; Tue, 25 Feb 2003 11:48:52 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA05752 for ; Tue, 25 Feb 2003 11:39:15 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PGm6p11018; Tue, 25 Feb 2003 11:48:06 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PGlDp10959 for ; Tue, 25 Feb 2003 11:47:13 -0500 Received: from mxic1.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA05709 for ; Tue, 25 Feb 2003 11:37:37 -0500 (EST) From: Black_David@emc.com Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1KDZ19CN>; Tue, 25 Feb 2003 11:41:06 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C93D@corpmx14.us.dg.com> To: Dwight.Barron@hp.com, rddp@ietf.org Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) Date: Tue, 25 Feb 2003 11:38:25 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Dwight, WG chair hat off ... > > > > I think if we are serious about supporting unmodified TCP, then > > > > the CRC should be optional. NFS typically runs with only TCP/UDP > > > > > > > I think there is a significant delta between "serious about > > > supporting" and "serious about optimizing for". > > > > An alternate viewpoint is that "serious about supporting" includes > > "not placing unreasonable implementation requirements upon". > > But unreasonable from which perspective. If we were assuming > symmetric implementations at each end, with roughly equal > costs for performing a function, then optionality can be > evaluated from end user trade-offs, usually functionality > (like CRC) vs. performance or cost to implement or whatever. > Asymmetric implementations mean that one end will (willingly) > bear more of a burden for the benefit of the other. I have > not heard nor seen any evidence that a CRC is an unreasonable > burden to implement in hardware. It is standard practice for > modern IETF protocols (SCTP), other modern storage and > datacenter interconnect technologies (FC, IB). I generally agree with much all of the above discussion, but the NFS example is conspicuous by its absence, as NFS is definitely a candidate ULP for RDDP. > Why do we want to burden over-burdened administrators with > yet another option (would you like data integrity with that?). That doesn't follow. A "mandatory to implement" requirement plus an implementation policy of "will not communicate with a peer who does not use CRCs" would avoid the administrative burden issue, and would be a good thing to describe as an acceptable policy. > The pervasive OSV support was exactly my point, unmodified > existing software stacks. However, I have been told, but have > not personally confirmed, that there is no easy/standard way > for a ULP above TCP to discover if IPSEC is actually being > invoked on a given connection with these existing stacks. If > this is actually the case, then exploiting overlapping > IPSEC/CRC coverage would seem to be left as yet another > exercise for the administrator. Or the configuration software that sets out to make the administrator's life easier by coordinating IPsec and RDDP configuration. > > And there are two scenarios discussed above where the CRC does not > > appear to be required. > > Again, these were/are very valid architectural decisions when > software implementations were heavily weighted in the > decision process. Hardware equivalent architectural decisions > (e.g. FC) that carry the exact same class of storage traffic > always have CRCs enabled. WG chair hat on for the following two paragraphs: RDDP is about more than just SCSI storage traffic. If RDDP were scoped only to IP Storage (and was in the IP Storage WG), this argument would carry much more significance. In case it's not clear, "mandatory to use" is a stiff requirement. The burden of demonstrating the need for such a requirement is generally on those advocating it. Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 25 11:40:46 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA05805 for ; Tue, 25 Feb 2003 11:40:45 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1PGnpI11111 for rddp-archive@odin.ietf.org; Tue, 25 Feb 2003 11:49:51 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PGnpp11108 for ; Tue, 25 Feb 2003 11:49:51 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA05790 for ; Tue, 25 Feb 2003 11:40:14 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PGn0p11093; Tue, 25 Feb 2003 11:49:00 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PGmnp11060 for ; Tue, 25 Feb 2003 11:48:49 -0500 Received: from maho3msx2.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA05749 for ; Tue, 25 Feb 2003 11:39:12 -0500 (EST) From: Black_David@emc.com Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1PRK6SCY>; Tue, 25 Feb 2003 11:42:21 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C93E@corpmx14.us.dg.com> To: cait@asomi.com, rddp@ietf.org Date: Tue, 25 Feb 2003 11:40:16 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] iSCSI-like markers (was Optional CRCs) Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > Specifically, would it be a valid goal to make DDP's > adaptation to TCP be compatible with iSCSI's? > > Compatible would not require them to be compatible. > For example, RDDP could require feature that iSCSI > considered to be optional. The benefits in logic > sharing would still apply. It could be a valid goal. Keep in mind that iSCSI markers have rarely (if ever) been observed in the wild ... Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 25 12:19:58 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA06968 for ; Tue, 25 Feb 2003 12:19:58 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1PHT5Y13920 for rddp-archive@odin.ietf.org; Tue, 25 Feb 2003 12:29:05 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PHT4p13917 for ; Tue, 25 Feb 2003 12:29:04 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA06961 for ; Tue, 25 Feb 2003 12:19:26 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PHSDp13867; Tue, 25 Feb 2003 12:28:13 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PHPep13752 for ; Tue, 25 Feb 2003 12:25:40 -0500 Received: from palrel12.hp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA06849 for ; Tue, 25 Feb 2003 12:16:02 -0500 (EST) Received: from esmail.cup.hp.com (esmail.cup.hp.com [15.0.65.164]) by palrel12.hp.com (Postfix) with ESMTP id 1FD8F1C0174D for ; Tue, 25 Feb 2003 09:19:57 -0800 (PST) Received: from mk731915.cup.hp.com (mk731915.cup.hp.com [15.8.80.111]) by esmail.cup.hp.com (8.9.3/8.8.6) with ESMTP id JAA26150 for ; Tue, 25 Feb 2003 09:17:11 -0800 (PST) Message-Id: <5.1.0.14.2.20030225070043.01ec4100@esmail.cup.hp.com> X-Sender: krause@esmail.cup.hp.com X-Mailer: QUALCOMM Windows Eudora Version 5.1 Date: Tue, 25 Feb 2003 07:03:07 -0800 To: rddp@ietf.org From: Michael Krause Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) In-Reply-To: <277DD60FB639D511AC0400B0D068B71E0564C93A@corpmx14.us.dg.co m> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii"; format=flowed Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , At 09:43 AM 2/25/2003 -0500, Black_David@emc.com wrote: >Dwight, > > > > I think if we are serious about supporting unmodified TCP, then > > > the CRC should be optional. NFS typically runs with only TCP/UDP > > > > > I think there is a significant delta between "serious about > > supporting" and "serious about optimizing for". > >An alternate viewpoint is that "serious about supporting" includes >"not placing unreasonable implementation requirements upon". NFS >as currently used does not need CRCs (although NFS did learn the >hard way that turning on checksums was a real good idea), and ... > > > > checksum for protection. Not a good idea to do this over the > > > public internet, but then critical data traveling through > > > the public internet will (or should) use IPSec protection. > > > > > And where does the IPSEC processing power come from? I would be > > willing to bet there is a large overlap of candidate environments > > that need to run RDDP in unmodified software stacks that will > > need to run IPSEC in that same unmodified software stack. IPSEC > > is great in its own right, but is certainly not a good tradeoff > > when a CRC would suffice. > >To begin with, there's quite a bit of OSV support for IPsec in >existing stacks today. The point here is not that IPsec is >computationally cheap (it's not), but rather that in an environment >where IPsec is required for security policy reasons and the IPsec >coverage is similar to the CRC coverage, the CRC may not be needed >as the 96-bit IPsec MAC provides a considerably stronger integrity >check - see Section 3.6 of draft-ietf-ips-security-19.txt. As >Jim Williams wrote: The reason I have heard quoted by many is the IPSec is often implemented in a gateway device and is therefore not end-to-end which is why end-to-end CRC is still required by many. I know from discussions with customers on the use of end-to-end CRC that none view it is as optional for iSCSI. Yes, for initial evaluation purposes with early technology they are willing to live with it but for actual production deployment, they are not willing to live without it on and IPSec is not a substitute as they are using VPN gateways which are relatively cheap to doing this in an end device. Mike > > > iSCSI is a precedent for optional CRCs. > >And there are two scenarios discussed above where the CRC does not >appear to be required. > >Thanks, >--David >---------------------------------------------------- >David L. Black, Senior Technologist >EMC Corporation, 176 South St., Hopkinton, MA 01748 >+1 (508) 293-7953 FAX: +1 (508) 293-7786 >black_david@emc.com Mobile: +1 (978) 394-7754 >---------------------------------------------------- >_______________________________________________ >rddp mailing list >rddp@ietf.org >https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 25 13:01:40 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA08732 for ; Tue, 25 Feb 2003 13:01:40 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1PIAkk18396 for rddp-archive@odin.ietf.org; Tue, 25 Feb 2003 13:10:46 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PIAkp18393 for ; Tue, 25 Feb 2003 13:10:46 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA08727 for ; Tue, 25 Feb 2003 13:01:09 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PIA4p18374; Tue, 25 Feb 2003 13:10:04 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PI9dp18316 for ; Tue, 25 Feb 2003 13:09:39 -0500 Received: from msgbas2x.cos.agilent.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA08668 for ; Tue, 25 Feb 2003 13:00:01 -0500 (EST) From: pat_thaler@agilent.com Received: from relcos2.cos.agilent.com (relcos2.cos.agilent.com [130.29.152.237]) by msgbas2x.cos.agilent.com (Postfix) with ESMTP id B63F519A6; Tue, 25 Feb 2003 11:03:54 -0700 (MST) Received: from axcsbh2.cos.agilent.com (axcsbh2.cos.agilent.com [130.29.152.144]) by relcos2.cos.agilent.com (Postfix) with SMTP id 84EDC601; Tue, 25 Feb 2003 11:03:50 -0700 (MST) Received: from 130.29.152.144 by axcsbh2.cos.agilent.com (InterScan E-Mail VirusWall NT); Tue, 25 Feb 2003 11:03:49 -0700 Received: by axcsbh2.cos.agilent.com with Internet Mail Service (5.5.2653.19) id <1RKLCSZA>; Tue, 25 Feb 2003 11:03:49 -0700 Message-ID: <1BEBA5E8600DD4119A50009027AF54A00131C8D7@axcs04.cos.agilent.com> To: Black_David@emc.com, krause@cup.hp.com, rddp@ietf.org Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) Date: Tue, 25 Feb 2003 11:03:46 -0700 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , David, The TCP alternative checksum of 1146 (Fletcher)would not have been an adequate alternate solution because it provides much weaker protection against undetected errors than CRC. It does provide some improvement over the protection of the normal TCP checksum but the improvement is only modest. I provided the analysis of Fletcher and Adler (another checksum alternative) to IPS reflector at the time this was discussed. The impact of the alternative checksums are summarized and compared to the protection of a CRC in RFC 3385. Pat -----Original Message----- From: Black_David@emc.com [mailto:Black_David@emc.com] Sent: Tuesday, February 25, 2003 6:26 AM To: krause@cup.hp.com; rddp@ietf.org Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) > Isn't SCTP CRC protection required per chunk? Actually, it's per packet (in the SCTP common header), as Caitlin has pointed out. SCTP did not have a checksum that needed to be preserved, in contrast to TCP. The IPS WG considered the TCP alternate checksum mechanism in RFC 1146 as a means of using a CRC to replace the TCP checksum, but that mechanism is undeployable in practice due to what it breaks, even though it would have been a cleaner solution in principle. > The reason to have a CRC for > MPA is to provide data integrity below DDP/RDMA / ULP which has been > demonstrated as a concern during the development of iSCSI. Do we really > need to go back through the entire IPS debate as to why CRCs are good > things here or accept that a CRC at or above the transport but below > DDP/RDMA / ULP is a good thing? We don't need to go back through that debate, because the question isn't about "good to implement" - it's about "mandatory to use", which is a very different issue. CRCs are optional for iSCSI, and NFS (another important client for RDDP) has never needed CRCs, as the TCP checksum has been adequate for its purposes (yes, a CRC would be an improvement). Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 25 13:12:38 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA09141 for ; Tue, 25 Feb 2003 13:12:38 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1PILin18972 for rddp-archive@odin.ietf.org; Tue, 25 Feb 2003 13:21:44 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PILip18969 for ; Tue, 25 Feb 2003 13:21:44 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA09108 for ; Tue, 25 Feb 2003 13:12:06 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PIL5p18933; Tue, 25 Feb 2003 13:21:05 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PIK7p18879 for ; Tue, 25 Feb 2003 13:20:07 -0500 Received: from dmz1.silverbacksystems.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA09045 for ; Tue, 25 Feb 2003 13:10:28 -0500 (EST) Received: from ns.silverbacksystems.com (gate-camp-hme0.silverbacksystems.com [65.172.158.93]) by dmz1.silverbacksystems.com (Postfix on SuSE Linux 7.3 (i386)) with ESMTP id F3650D157; Tue, 25 Feb 2003 10:14:09 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with ESMTP id A765919A78; Tue, 25 Feb 2003 10:14:09 -0800 (PST) Content-Transfer-Encoding: 7bit Content-Type: text/plain; charset="iso-8859-1" Date: Tue, 25 Feb 2003 10:13:25 -0800 From: "Somesh Gupta" Importance: Normal In-Reply-To: <01CA1B1F74A4EC40A9C887F7B58F8724031430E6@cceexc16.americas.cpqcorp.net> Message-ID: MIME-Version: 1.0 Received: from ns.silverbacksystems.com (localhost [127.0.0.1]) by localhost (AvMailGate-2.0.0.6) id 14240-7E5B5545; Tue, 25 Feb 2003 10:13:13 -0800 Received: from somesh (somesh.corp.silverbacksystems.com [10.0.16.35]) by ns.silverbacksystems.com (Postfix on SuSE Linux eMail Server 3.0) with SMTP id 4FC2919940; Tue, 25 Feb 2003 10:13:13 -0800 (PST) Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) To: "Barron, Dwight" , , X-AntiVirus: OK! AntiVir MailGate Version 2.0.0.6 at ns.silverbacksystems.com has not found any known virus in this email. X-Mailer: Microsoft Outlook IMO, Build 9.0.2416 (9.0.2910.0) X-MimeOLE: Produced By Microsoft MimeOLE V5.00.2919.6700 X-MSMail-Priority: Normal X-Priority: 3 (Normal) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Dwight, We went through some similar debates in iSCSI with one camp assuming primarily hardware based initiators and targets, and the other group assuming a number of software based implementations (I started off assuming hardware based but switched later). As it turns out, the adoption is being driven by software based initiators and it is a good thing that the optionality allowed software based initiators to avoid the features that would have made this avenue futile. Perhaps there would not be so much debate if RDMA was focused on IPC in the data center. But with storage being a primary apps, it is important to enable this adoption route. Negotiated options seem like a good engineering choice. Somesh > -----Original Message----- > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > Barron, Dwight > Sent: Tuesday, February 25, 2003 7:58 AM > To: Black_David@emc.com; rddp@ietf.org > Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) > > > > > > -----Original Message----- > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > Sent: Tuesday, February 25, 2003 8:44 AM > > To: Barron, Dwight; rddp@ietf.org > > Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) > > > > > > Dwight, > > > > > > I think if we are serious about supporting unmodified TCP, then > > > > the CRC should be optional. NFS typically runs with only TCP/UDP > > > > > > > I think there is a significant delta between "serious about > > > supporting" and "serious about optimizing for". > > > > An alternate viewpoint is that "serious about supporting" includes > > "not placing unreasonable implementation requirements upon". > > But unreasonable from which perspective. If we were assuming > symmetric implementations at each end, with roughly equal costs > for performing a function, then optionality can be evaluated from > end user trade-offs, usually functionality (like CRC) vs. > performance or cost to implement or whatever. Asymmetric > implementations mean that one end will (willingly) bear more of a > burden for the benefit of the other. I have not heard nor seen > any evidence that a CRC is an unreasonable burden to implement in > hardware. It is standard practice for modern IETF protocols > (SCTP), other modern storage and datacenter interconnect > technologies (FC, IB). Why do we want to burden over-burdened > administrators with yet another option (would you like data > integrity with that?). Note that I am not trying to be > unsympathetic to the needs and desires of efficient software > implementations, I am trying to balance the realities asymmetric > implementations and the need for optionality, both from an implement! > ation/test perspective and the end user administrator. > > > > NFS > > as currently used does not need CRCs (although NFS did learn the > > hard way that turning on checksums was a real good idea), and ... > > > > > > checksum for protection. Not a good idea to do this over the > > > > public internet, but then critical data traveling through > > > > the public internet will (or should) use IPSec protection. > > > > > > > And where does the IPSEC processing power come from? I would be > > > willing to bet there is a large overlap of candidate environments > > > that need to run RDDP in unmodified software stacks that will > > > need to run IPSEC in that same unmodified software stack. IPSEC > > > is great in its own right, but is certainly not a good tradeoff > > > when a CRC would suffice. > > > > To begin with, there's quite a bit of OSV support for IPsec in > > existing stacks today. The point here is not that IPsec is > > computationally cheap (it's not), but rather that in an environment > > where IPsec is required for security policy reasons and the IPsec > > coverage is similar to the CRC coverage, the CRC may not be needed > > as the 96-bit IPsec MAC provides a considerably stronger integrity > > check - see Section 3.6 of draft-ietf-ips-security-19.txt. As > > The pervasive OSV support was exactly my point, unmodified > existing software stacks. However, I have been told, but have not > personally confirmed, that there is no easy/standard way for a > ULP above TCP to discover if IPSEC is actually being invoked on a > given connection with these existing stacks. If this is actually > the case, then exploiting overlapping IPSEC/CRC coverage would > seem to be left as yet another exercise for the administrator. > > > > Jim Williams wrote: > > > > > > iSCSI is a precedent for optional CRCs. > > > > And there are two scenarios discussed above where the CRC does not > > appear to be required. > > Again, these were/are very valid architectural decisions when > software implementations were heavily weighted in the decision > process. Hardware equivalent architectural decisions (e.g. FC) > that carry the exact same class of storage traffic always have > CRCs enabled. > > > > > Thanks, > > --David > > ---------------------------------------------------- > > David L. Black, Senior Technologist > > EMC Corporation, 176 South St., Hopkinton, MA 01748 > > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > > black_david@emc.com Mobile: +1 (978) 394-7754 > > ---------------------------------------------------- > > > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 25 13:23:39 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA09559 for ; Tue, 25 Feb 2003 13:23:39 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1PIWj719460 for rddp-archive@odin.ietf.org; Tue, 25 Feb 2003 13:32:45 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PIWjp19457 for ; Tue, 25 Feb 2003 13:32:45 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA09549 for ; Tue, 25 Feb 2003 13:23:08 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PIW4p19406; Tue, 25 Feb 2003 13:32:04 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PIVZp19389 for ; Tue, 25 Feb 2003 13:31:35 -0500 Received: from mxic1.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA09524 for ; Tue, 25 Feb 2003 13:21:58 -0500 (EST) From: Black_David@emc.com Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1KDZF1SM>; Tue, 25 Feb 2003 13:25:23 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C942@corpmx14.us.dg.com> To: pat_thaler@agilent.com, rddp@ietf.org Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) Date: Tue, 25 Feb 2003 13:22:42 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Pat, I completely agree about the strength of Fletcher checksums - what I was intending to refer to was a potential use of RFC 1146's mechanisms to replace the TCP checksum with a real CRC. This would have involved defining a new chksum value for the CRC, but that isn't practical because when there isn't a TCP checksum in the TCP checksum field, too much stuff breaks. This idea was rejected out-of-hand after being briefly considered. Thanks, --David > -----Original Message----- > From: pat_thaler@agilent.com [mailto:pat_thaler@agilent.com] > Sent: Tuesday, February 25, 2003 1:04 PM > To: Black_David@emc.com; krause@cup.hp.com; rddp@ietf.org > Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) > > > David, > > The TCP alternative checksum of 1146 (Fletcher)would not have > been an adequate alternate solution because it provides much > weaker protection against undetected errors than CRC. It does > provide some improvement over the protection of the normal > TCP checksum but the improvement is only modest. I provided > the analysis of Fletcher and Adler (another checksum > alternative) to IPS reflector at the time this was discussed. > The impact of the alternative checksums are summarized and > compared to the protection of a CRC in RFC 3385. > > Pat > > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Tuesday, February 25, 2003 6:26 AM > To: krause@cup.hp.com; rddp@ietf.org > Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) > > > > Isn't SCTP CRC protection required per chunk? > > Actually, it's per packet (in the SCTP common header), as Caitlin has > pointed out. SCTP did not have a checksum that needed to be > preserved, > in contrast to TCP. The IPS WG considered the TCP alternate checksum > mechanism in RFC 1146 as a means of using a CRC to replace the TCP > checksum, but that mechanism is undeployable in practice due to what > it breaks, even though it would have been a cleaner solution > in principle. > > > The reason to have a CRC for > > MPA is to provide data integrity below DDP/RDMA / ULP > which has been > > demonstrated as a concern during the development of iSCSI. > Do we really > > need to go back through the entire IPS debate as to why > CRCs are good > > things here or accept that a CRC at or above the transport > but below > > DDP/RDMA / ULP is a good thing? > > We don't need to go back through that debate, because the question > isn't about "good to implement" - it's about "mandatory to use", which > is a very different issue. CRCs are optional for iSCSI, and > NFS (another > important client for RDDP) has never needed CRCs, as the TCP checksum > has been adequate for its purposes (yes, a CRC would be an > improvement). > > Thanks, > --David > ---------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > black_david@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Tue Feb 25 15:10:48 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA13294 for ; Tue, 25 Feb 2003 15:10:48 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1PKJvX27269 for rddp-archive@odin.ietf.org; Tue, 25 Feb 2003 15:19:57 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PKJvp27266 for ; Tue, 25 Feb 2003 15:19:57 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA13275 for ; Tue, 25 Feb 2003 15:10:16 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PKJDp27239; Tue, 25 Feb 2003 15:19:13 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1PKFWp27122 for ; Tue, 25 Feb 2003 15:15:32 -0500 Received: from ztxmail04.ztx.compaq.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA13120 for ; Tue, 25 Feb 2003 15:05:51 -0500 (EST) Received: from cceexg11.americas.cpqcorp.net (cceexg11.americas.cpqcorp.net [16.110.250.125]) by ztxmail04.ztx.compaq.com (Postfix) with ESMTP id EA5D34DF7; Tue, 25 Feb 2003 14:09:44 -0600 (CST) Received: from cceexc16.americas.cpqcorp.net ([16.110.250.58]) by cceexg11.americas.cpqcorp.net with Microsoft SMTPSVC(5.0.2195.2966); Tue, 25 Feb 2003 14:09:44 -0600 X-MimeOLE: Produced By Microsoft Exchange V6.0.6375.0 content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) Date: Tue, 25 Feb 2003 14:09:43 -0600 Message-ID: <01CA1B1F74A4EC40A9C887F7B58F8724031430E7@cceexc16.americas.cpqcorp.net> Thread-Topic: Optional CRCs ( was [rddp] Marker Interval ) Thread-Index: AcLc+cLNsFw9qjZ2RJGgyI+hC8OhNgACP0RQ From: "Barron, Dwight" To: "Somesh Gupta" , X-OriginalArrivalTime: 25 Feb 2003 20:09:44.0828 (UTC) FILETIME=[D6D14BC0:01C2DD09] Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by www1.ietf.org id h1PKFWp27123 Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 8bit Content-Transfer-Encoding: 8bit Somesh, I am not discounting the importance of software implementations to enable RDMA, even in the datacenter. What I am questioning is the supposition that CRC is so performance prohibitive that it is "placing unreasonable implementation requirements upon" software stack implementations and therefore needs to be optional. The SCTP working group apparently came to the conclusion it does not and therefore made CRCs mandatory. To my knowledge there have not been any complaints on this reflector that SCTP's CRC makes RDDP/SCTP unreasonable to implement in software. While one may claim that an RDDP/TCP CRC is superfluous or unnecessary, I contend that the SCTP decision is still precedent for the unreasonable test. My personal observations/experience [your mileage may vary caveat goes here] with iSCSI suggests that the optional CRC didn't buy the "unmodified software stack" implementations as much benefit as might be expected. The software implementations ended up doing buffer copies after all (which means computing a CRC wouldn't have cost that much more) and they are being targeted at environments where there are either excess CPU cycles available or the storage traffic overhead is only accounting for 10-20% of the CPU. Clearly there are software implementations that are using SGLs to avoid copying, but I don't think they fall into the "unmodified software stack" category (either user space or kernel mode access to the network stack), which I believe was a fundamental tenet of this thread. Regards, Dwight > -----Original Message----- > From: Somesh Gupta [mailto:somesh_gupta@silverbacksystems.com] > Sent: Tuesday, February 25, 2003 12:13 PM > To: Barron, Dwight; Black_David@emc.com; rddp@ietf.org > Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) > > > Dwight, > > We went through some similar debates in iSCSI with one camp > assuming primarily hardware based initiators and targets, > and the other group assuming a number of software based > implementations (I started off assuming hardware based but > switched later). > > As it turns out, the adoption is being driven by software > based initiators and it is a good thing that the optionality > allowed software based initiators to avoid the features > that would have made this avenue futile. > > Perhaps there would not be so much debate if RDMA was focused > on IPC in the data center. But with storage being a primary apps, > it is important to enable this adoption route. Negotiated > options seem like a good engineering choice. > > Somesh > > > -----Original Message----- > > From: rddp-admin@ietf.org [mailto:rddp-admin@ietf.org]On Behalf Of > > Barron, Dwight > > Sent: Tuesday, February 25, 2003 7:58 AM > > To: Black_David@emc.com; rddp@ietf.org > > Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) > > > > > > > > > > > -----Original Message----- > > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > > Sent: Tuesday, February 25, 2003 8:44 AM > > > To: Barron, Dwight; rddp@ietf.org > > > Subject: RE: Optional CRCs ( was [rddp] Marker Interval ) > > > > > > > > > Dwight, > > > > > > > > I think if we are serious about supporting unmodified > TCP, then > > > > > the CRC should be optional. NFS typically runs with > only TCP/UDP > > > > > > > > > I think there is a significant delta between "serious about > > > > supporting" and "serious about optimizing for". > > > > > > An alternate viewpoint is that "serious about supporting" includes > > > "not placing unreasonable implementation requirements upon". > > > > But unreasonable from which perspective. If we were assuming > > symmetric implementations at each end, with roughly equal costs > > for performing a function, then optionality can be evaluated from > > end user trade-offs, usually functionality (like CRC) vs. > > performance or cost to implement or whatever. Asymmetric > > implementations mean that one end will (willingly) bear more of a > > burden for the benefit of the other. I have not heard nor seen > > any evidence that a CRC is an unreasonable burden to implement in > > hardware. It is standard practice for modern IETF protocols > > (SCTP), other modern storage and datacenter interconnect > > technologies (FC, IB). Why do we want to burden over-burdened > > administrators with yet another option (would you like data > > integrity with that?). Note that I am not trying to be > > unsympathetic to the needs and desires of efficient software > > implementations, I am trying to balance the realities asymmetric > > implementations and the need for optionality, both from an > implement! > > ation/test perspective and the end user administrator. > > > > > > > NFS > > > as currently used does not need CRCs (although NFS did learn the > > > hard way that turning on checksums was a real good idea), and ... > > > > > > > > checksum for protection. Not a good idea to do this over the > > > > > public internet, but then critical data traveling through > > > > > the public internet will (or should) use IPSec protection. > > > > > > > > > And where does the IPSEC processing power come from? I would be > > > > willing to bet there is a large overlap of candidate > environments > > > > that need to run RDDP in unmodified software stacks that will > > > > need to run IPSEC in that same unmodified software stack. IPSEC > > > > is great in its own right, but is certainly not a good tradeoff > > > > when a CRC would suffice. > > > > > > To begin with, there's quite a bit of OSV support for IPsec in > > > existing stacks today. The point here is not that IPsec is > > > computationally cheap (it's not), but rather that in an > environment > > > where IPsec is required for security policy reasons and the IPsec > > > coverage is similar to the CRC coverage, the CRC may not be needed > > > as the 96-bit IPsec MAC provides a considerably stronger integrity > > > check - see Section 3.6 of draft-ietf-ips-security-19.txt. As > > > > The pervasive OSV support was exactly my point, unmodified > > existing software stacks. However, I have been told, but have not > > personally confirmed, that there is no easy/standard way for a > > ULP above TCP to discover if IPSEC is actually being invoked on a > > given connection with these existing stacks. If this is actually > > the case, then exploiting overlapping IPSEC/CRC coverage would > > seem to be left as yet another exercise for the administrator. > > > > > > > Jim Williams wrote: > > > > > > > > iSCSI is a precedent for optional CRCs. > > > > > > And there are two scenarios discussed above where the CRC does not > > > appear to be required. > > > > Again, these were/are very valid architectural decisions when > > software implementations were heavily weighted in the decision > > process. Hardware equivalent architectural decisions (e.g. FC) > > that carry the exact same class of storage traffic always have > > CRCs enabled. > > > > > > > > Thanks, > > > --David > > > ---------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 176 South St., Hopkinton, MA 01748 > > > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > ---------------------------------------------------- > > > > > _______________________________________________ > > rddp mailing list > > rddp@ietf.org > > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Feb 26 10:20:45 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA10677 for ; Wed, 26 Feb 2003 10:20:45 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1QFUGQ17316 for rddp-archive@odin.ietf.org; Wed, 26 Feb 2003 10:30:16 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1QFUGp17313 for ; Wed, 26 Feb 2003 10:30:16 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA10661 for ; Wed, 26 Feb 2003 10:20:12 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1QFP9p17087; Wed, 26 Feb 2003 10:25:09 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1QFOCp16988 for ; Wed, 26 Feb 2003 10:24:12 -0500 Received: from mx01.netapp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA10277 for ; Wed, 26 Feb 2003 10:14:07 -0500 (EST) Received: from frejya.corp.netapp.com (frejya [10.10.20.91]) by mx01.netapp.com (8.12.6/8.12.6/NTAP-1.4) with ESMTP id h1QFI2H2001459 for ; Wed, 26 Feb 2003 07:18:02 -0800 (PST) Received: from tahoe.corp.netapp.com (tahoe.corp.netapp.com [10.10.22.112]) by frejya.corp.netapp.com (8.12.6/8.12.6/NTAP-1.4) with ESMTP id h1QFI2M4008145 for ; Wed, 26 Feb 2003 07:18:02 -0800 (PST) Received: from eisler.com (10.58.48.132 [10.58.48.132]) by tahoe.corp.netapp.com with SMTP (Microsoft Exchange Internet Mail Service Version 5.5.2650.21) id 1JT75LDX; Wed, 26 Feb 2003 07:17:21 -0800 Message-ID: <3E5CDA9D.5020403@eisler.com> Date: Wed, 26 Feb 2003 07:17:49 -0800 From: Mike Eisler User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:0.9.4.1) Gecko/20020508 Netscape6/6.2.3 X-Accept-Language: en-us MIME-Version: 1.0 To: rddp@ietf.org Subject: Re: Optional CRCs ( was [rddp] Marker Interval ) Content-Type: text/plain; charset=us-ascii; format=flowed Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit >> An alternate viewpoint is that "serious about supporting" includes >> "not placing unreasonable implementation requirements upon". NFS >> as currently used does not need CRCs (although NFS did learn the >> hard way that turning on checksums was a real good idea), and ... > >Just so no one is confused, while NFS typically runs over >TCP or UDP using their default checksums and most find that >an adequate solution, NFS also can optionally use RPCSEC_GSS >to add integrity checking if desired. So NFS has belts and >suspenders, although most just wear a belt. Adding yet another >integrity check is not required for NFS. Violent agreement here. It is the duty of the transport to ensure reliable delivery, not NFS. However, while RPCSEC_GSS indeed provides integrity checking, if the server fails to verify the arguments of the remote procedure call, an error is returned to the caller. The caller normally treats this error as a server side "xdr decode error", and the error is propogated to the application. The rationale is if a message passed the transport level checksum, but failed the crypto-level checksum, then an attacker is clearly trying to tamper with the traffic. One could certainly extend the semantics of RPCSEC_GSS to CRC-style checksumming that would result in the RPC-level of the stack re-transmitting. But that semantic is not currently specified (and as I've noted above, a layering violation, imho). -mre _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Wed Feb 26 17:24:15 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA29733 for ; Wed, 26 Feb 2003 17:24:15 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1QMXvQ16956 for rddp-archive@odin.ietf.org; Wed, 26 Feb 2003 17:33:57 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1QMXvp16953 for ; Wed, 26 Feb 2003 17:33:57 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA29684 for ; Wed, 26 Feb 2003 17:23:44 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1QMXDp16914; Wed, 26 Feb 2003 17:33:13 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1QMWdp16866 for ; Wed, 26 Feb 2003 17:32:39 -0500 Received: from mxic1.corp.emc.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA29622 for ; Wed, 26 Feb 2003 17:22:25 -0500 (EST) From: Black_David@emc.com Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id <1KDZHMDZ>; Wed, 26 Feb 2003 17:25:55 -0500 Message-ID: <277DD60FB639D511AC0400B0D068B71E0564C95C@corpmx14.us.dg.com> To: rddp@ietf.org Subject: RE: [rddp] SF RDDP meeting time change Date: Wed, 26 Feb 2003 17:23:15 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Everyone, The longer RDDP session was moved again - instead of Monday, it's now 2 hours on Thursday afternoon: THURSDAY, March 20, 2003 1530-1730 Afternoon Sessions II APP lemonade License to Enhance Messaging Oriented Network Access for Diverse Endpoints PWG INT eap Extensible Authentication Protocol WG RTG vrrp Virtual Router Redundancy Protocol WG OPS ipfix IP Flow Information Export WG SEC saag Open Security Area Directorate TSV pads & Path-Decoupled Signaling BOF & plpmtud Packetization Layer Path MTU Discovery BOF TSV rddp Remote Direct Data Placement WG I think this is final, but double-check the IETF web site tomorrow and/or Friday to be sure. The 1 hour Tuesday session has not changed. Thanks, --David > -----Original Message----- > From: Black_David@emc.com [mailto:Black_David@emc.com] > Sent: Monday, February 24, 2003 5:32 PM > To: rddp@ietf.org > Subject: [rddp] SF RDDP meeting time change > Importance: High > > > For those tracking the RDDP meeting times in San Francisco, > the 2 hour meeting slot for RDDP just turned into a 2.5 hour > slot and moved to Monday. This is not on the IETF web site > yet, but will be soon, and is typical of the sort of agenda > juggling that takes place before the IETF meeting times > are finalized. I apologize if I haven't issued warnings > that these meeting times do shift. > > Thanks, > --David > > MONDAY, March 17, 2003 > 0900-1130 Morning Sessions > APP apparea Applications Open Area Meeting > GEN ipr Intellectual Property Rights WG > INT dhc Dynamic Host Configuration WG > SUB ccamp Common Control and Measurement Plane WG > TSV nsis Next Steps in Signaling WG > TSV rddp Remote Direct Data Placement WG > > TUESDAY, March 18, 2003 > 1700-1800 Afternoon Sessions IV > APP lemonade License to Enhance Messaging Oriented > Network Access for > Diverse Endpoints PWG > APP webdav WWW Distributed Authoring and Versioning WG > INT itrace ICMP Traceback WG > OPS grow Global Routing Operations BOF > SEC ipseckey IPSEC KEYing information resource record WG > TSV iptel IP Telephony WG > TSV rddp Remote Direct Data Placement WG > > > ---------------------------------------------------- > David L. Black, Senior Technologist > EMC Corporation, 176 South St., Hopkinton, MA 01748 > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > black_david@emc.com Mobile: +1 (978) 394-7754 > ---------------------------------------------------- > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp > _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Thu Feb 27 07:49:45 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA28263 for ; Thu, 27 Feb 2003 07:49:45 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1RCxjZ15062 for rddp-archive@odin.ietf.org; Thu, 27 Feb 2003 07:59:45 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1RCxjp15059 for ; Thu, 27 Feb 2003 07:59:45 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA28219 for ; Thu, 27 Feb 2003 07:49:13 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1RCx4p14977; Thu, 27 Feb 2003 07:59:04 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1RCtip14688 for ; Thu, 27 Feb 2003 07:55:44 -0500 Received: from CNRI.Reston.VA.US (localhost [127.0.0.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id HAA27822; Thu, 27 Feb 2003 07:45:13 -0500 (EST) Message-Id: <200302271245.HAA27822@ietf.org> Mime-Version: 1.0 Content-Type: Multipart/Mixed; Boundary="NextPart" To: IETF-Announce: ; Cc: rddp@ietf.org From: Internet-Drafts@ietf.org Reply-to: Internet-Drafts@ietf.org Date: Thu, 27 Feb 2003 07:45:13 -0500 Subject: [rddp] I-D ACTION:draft-stewart-rddp-sctp-02.txt Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , --NextPart A New Internet-Draft is available from the on-line Internet-Drafts directories. This draft is a work item of the Remote Direct Data Placement Working Group of the IETF. Title : Stream Control Transmission Protocol (SCTP) Remote Direct Memory Access (RDMA) Direct Data Placement (DDP) Adaption Author(s) : R. Stewart et al. Filename : draft-stewart-rddp-sctp-02.txt Pages : 18 Date : 2003-2-26 This document describes a method to adapt Direct Data Placement (DDP) and Remote Direct Memory Access (RDMA) to Stream Control Transmission Protocol (SCTP) RFC2960 [2] using a generic description found in [RDMA-Draft] [4] and [DDP-Draft] [3]. A URL for this Internet-Draft is: http://www.ietf.org/internet-drafts/draft-stewart-rddp-sctp-02.txt To remove yourself from the IETF Announcement list, send a message to ietf-announce-request with the word unsubscribe in the body of the message. Internet-Drafts are also available by anonymous FTP. Login with the username "anonymous" and a password of your e-mail address. After logging in, type "cd internet-drafts" and then "get draft-stewart-rddp-sctp-02.txt". A list of Internet-Drafts directories can be found in http://www.ietf.org/shadow.html or ftp://ftp.ietf.org/ietf/1shadow-sites.txt Internet-Drafts can also be obtained by e-mail. Send a message to: mailserv@ietf.org. In the body type: "FILE /internet-drafts/draft-stewart-rddp-sctp-02.txt". NOTE: The mail server at ietf.org can return the document in MIME-encoded form by using the "mpack" utility. To use this feature, insert the command "ENCODING mime" before the "FILE" command. To decode the response(s), you will need "munpack" or a MIME-compliant mail reader. Different MIME-compliant mail readers exhibit different behavior, especially when dealing with "multipart" MIME messages (i.e. documents which have been split up into multiple messages), so check your local documentation on how to manipulate these messages. Below is the data which will enable a MIME compliant mail reader implementation to automatically retrieve the ASCII version of the Internet-Draft. --NextPart Content-Type: Multipart/Alternative; Boundary="OtherAccess" --OtherAccess Content-Type: Message/External-body; access-type="mail-server"; server="mailserv@ietf.org" Content-Type: text/plain Content-ID: <2003-2-26160917.I-D@ietf.org> ENCODING mime FILE /internet-drafts/draft-stewart-rddp-sctp-02.txt --OtherAccess Content-Type: Message/External-body; name="draft-stewart-rddp-sctp-02.txt"; site="ftp.ietf.org"; access-type="anon-ftp"; directory="internet-drafts" Content-Type: text/plain Content-ID: <2003-2-26160917.I-D@ietf.org> --OtherAccess-- --NextPart-- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 28 11:42:40 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA04178 for ; Fri, 28 Feb 2003 11:42:40 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1SGrCd10256 for rddp-archive@odin.ietf.org; Fri, 28 Feb 2003 11:53:12 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1SGrCp10253 for ; Fri, 28 Feb 2003 11:53:12 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA04174 for ; Fri, 28 Feb 2003 11:42:08 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1SGqKp10195; Fri, 28 Feb 2003 11:52:20 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1SGpop10170 for ; Fri, 28 Feb 2003 11:51:50 -0500 Received: from emulex.emulex.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA04117 for ; Fri, 28 Feb 2003 11:40:46 -0500 (EST) Received: from xbl.ad.emulex.com (xbl.ma.emulex.com [172.16.12.11]) by emulex.emulex.com (8.9.1a/8.9.1) with ESMTP id IAA12786 for ; Fri, 28 Feb 2003 08:44:41 -0800 (PST) Received: by xbl.ma.emulex.com with Internet Mail Service (5.5.2653.19) id <105KXF8L>; Fri, 28 Feb 2003 11:44:41 -0500 Message-ID: <3356669BBE90C448AD4645C843E2BF289B9007@xbl.ma.emulex.com> From: "Williams, Jim" To: "'rddp@ietf.org'" Date: Fri, 28 Feb 2003 11:44:40 -0500 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain; charset="iso-8859-1" Subject: [rddp] SCTP mapping Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Regarding http://www.ietf.org/internet-drafts/draft-stewart-rddp-sctp-02.txt Looks like a good, well thought out proposal. I have a couple points. ----- In section 3.2, I would prefer to see a bit more explicit statement something like the following. DDP-enabled SCTP endpoints MAY refuse to connect with any remote endpoint that is multi-homed. If a multi-homed DDP-enabled SCTP endpoint attemts to connect, and the connection is rejected with a resource error, then the connection SHOULD be reattempted with fewer specified IP addresses, and ultimately in non-multi homed mode. My sense is that this is the intent anyway, so I am only hoping for a bit more explicit language. ----- In section 4, it is stated, "DDP should request the maximum [number or streams per association] it will wish to use from SCTP." I would encourage a cleaner layering here. DDP should not need to be aware that it is layered on SCTP, and should not be concerned with how SCTP allocates DDP streams among SCTP associations. I am guessing that most implementations will chose to support one stream per association because it is simpler and I don't see a performance down side. However since number of streams per association is negotiable, it is fine to allow more. I am in favor, though, of keeping the complexity of deciding the desired number of streams per association, and negotiating with the remote end point the actual number supported layered below the DDP layer. DDP does not and should not care if two of its streams are in the same or in different SCTP associations. But I don't claim to have thought this through completely, and so am quite willing to change my mind in the face of good arguments. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 28 12:08:12 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA05005 for ; Fri, 28 Feb 2003 12:08:12 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1SHIjV12829 for rddp-archive@odin.ietf.org; Fri, 28 Feb 2003 12:18:45 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1SHIjp12826 for ; Fri, 28 Feb 2003 12:18:45 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA05002 for ; Fri, 28 Feb 2003 12:07:40 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1SHI3p12805; Fri, 28 Feb 2003 12:18:03 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1SHH8p12755 for ; Fri, 28 Feb 2003 12:17:08 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA04946 for ; Fri, 28 Feb 2003 12:06:04 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id D0FA42456A3; Fri, 28 Feb 2003 12:09:59 -0500 (EST) Date: Fri, 28 Feb 2003 11:11:36 -0600 From: Caitlin Bestler Subject: Re: [rddp] SCTP mapping - 3.2 remote multi-homing. To: "Williams, Jim" Cc: "'rddp@ietf.org'" X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B9007@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/28/03, Williams, Jim wrote: >In section 3.2, I would prefer to see a bit more explicit >statement something like the following. > > DDP-enabled SCTP endpoints MAY refuse to connect with any > remote endpoint that is multi-homed. If a multi-homed > DDP-enabled SCTP endpoint attemts to connect, and the > connection is rejected with a resource error, then the > connection SHOULD be reattempted with fewer specified IP > addresses, and ultimately in non-multi homed mode. > >My sense is that this is the intent anyway, so I am only >hoping for a bit more explicit language. Allowing the remote end to be multi-homed has no impact upon RDDP. It does have some impact on the SCTP layer. I believe that you could argue that any SCTP endpoint MAY refuse to connect with a multi-homed peer by returning a resource error. However I would argue that it SHOULD NOT do so. The difference between "MAY refuse" and "SHOULD NOT refuse" is only one of shading though. The more germane point is that unless there are DDP layer reasons for refusing, I do not believe it belongs in the draft. Note that in contrast, *local* multi-homing has *major* impacts on rddp. The draft makes it clear that nobody is obligated to make their RDDP/SCTP endpoint multihomed, and in fact they SHOULD NOT do so by default if it impacts performance. Short of designing RDDP into the motherboard itself I do not think a multihomed RDDP/SCTP endpoint could ever match the performance of an RNIC-specific RDDP/SCTP endpoint. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 28 12:26:09 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA05747 for ; Fri, 28 Feb 2003 12:26:09 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1SHah713637 for rddp-archive@odin.ietf.org; Fri, 28 Feb 2003 12:36:43 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1SHahp13634 for ; Fri, 28 Feb 2003 12:36:43 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA05684 for ; Fri, 28 Feb 2003 12:25:36 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1SHa1p13595; Fri, 28 Feb 2003 12:36:01 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1SHZRp13575 for ; Fri, 28 Feb 2003 12:35:27 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA05541 for ; Fri, 28 Feb 2003 12:24:22 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 9BCA1244EE5; Fri, 28 Feb 2003 12:28:17 -0500 (EST) Date: Fri, 28 Feb 2003 11:29:53 -0600 From: Caitlin Bestler Subject: Re: [rddp] SCTP mapping - Multiple Streams To: "Williams, Jim" Cc: "'rddp@ietf.org'" X-Priority: 3 In-Reply-To: <3356669BBE90C448AD4645C843E2BF289B9007@xbl.ma.emulex.com> Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/28/03, Williams, Jim wrote: >In section 4, it is stated, "DDP should request the >maximum [number or streams per association] it will wish >to use from SCTP." > >I would encourage a cleaner layering here. DDP should not >need to be aware that it is layered on SCTP, and should >not be concerned with how SCTP allocates DDP streams among >SCTP associations. > But DDP has to interface with the actual LLP. It is indeed good design to isolate the LLP dependencies, but ultimately whatever sub-component that does that isolation has to make requests of the actual LLP on terms that it understands. For SCTP the proposed mapping takes the assumption that the SCTP layer overhead for unordered streams is almost nil. Therefore the DDP layer should avoid making any advance commitment that would force it to create a second association at a future data when there was no need for it to do so. A second association can is also not possible for peer-to-peer applications that pre-specifiy the port number of both ends. In the more typical active/passive case only the passive side port number is fixed in advance, so additional associations are less of a problem, but still undesirable forthe reasons cited below. >I am guessing that most implementations will chose to >support one stream per association because it is simpler >and I don't see a performance down side. However since >number of streams per association is negotiable, it is >fine to allow more. On the contrary, there are major advantages to sharing the same association across multiple DDP streams consolidates acking, congestion control and setup. An integrated RDDP/SCTP implementation SHOULD NOT presume there is only one stream per association unless it has very strong reasons to believe that there will only be a single stream between the two endpoints. There is actually little reason to do a fully integrated RDDP/SCTP implementation. You can achieve virtually all ofthe performance benefits by leaving them cleanly layered but having a specialized SCTP layer that only supports unordered delivery. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 28 13:00:29 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA07493 for ; Fri, 28 Feb 2003 13:00:28 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1SIB3E16757 for rddp-archive@odin.ietf.org; Fri, 28 Feb 2003 13:11:03 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1SIB3p16754 for ; Fri, 28 Feb 2003 13:11:03 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA07459 for ; Fri, 28 Feb 2003 12:59:57 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1SIA8p16671; Fri, 28 Feb 2003 13:10:08 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1SI9kp16633 for ; Fri, 28 Feb 2003 13:09:46 -0500 Received: from thebe.your-site.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id MAA07433 for ; Fri, 28 Feb 2003 12:58:40 -0500 (EST) Received: from 192.168.0.2 (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 9DFD8244878; Fri, 28 Feb 2003 13:02:36 -0500 (EST) Date: Fri, 28 Feb 2003 12:04:12 -0600 From: Caitlin Bestler Subject: Re: [rddp] SCTP mapping - Multiple Streams To: "'rddp@ietf.org'" Cc: "Williams, Jim" X-Priority: 3 In-Reply-To: Message-ID: MIME-Version: 1.0 Content-Type: text/plain; Charset=US-ASCII Content-Transfer-Encoding: 7bit X-Mailer: Mailsmith 1.5.3 (Blindsider) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On 2/28/03, Caitlin Bestler wrote: > >For SCTP the proposed mapping takes the assumption that the >SCTP layer overhead for unordered streams is almost nil. >Therefore the DDP layer should avoid making any advance >commitment that would force it to create a second >association at a future data when there was no need for it >to do so. > But that makes me realize that the next revision should have the following language added: DDP exclusively uses unordered Data Chunks. The SCTP layer may be able to optimize the resources allocated to support an SCTP stream if advised that a stream will never be used for ordered Data Chunks. If a local interface is available to provide such information it MUST be used. Caitlin Bestler http://asomi.com/CaitlinBestler/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From mailnull@www1.ietf.org Fri Feb 28 15:42:17 2003 Received: from www1.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA12633 for ; Fri, 28 Feb 2003 15:42:16 -0500 (EST) Received: (from mailnull@localhost) by www1.ietf.org (8.11.6/8.11.6) id h1SKotf26874 for rddp-archive@odin.ietf.org; Fri, 28 Feb 2003 15:50:55 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1SKotp26871 for ; Fri, 28 Feb 2003 15:50:55 -0500 Received: from www1.ietf.org (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA12618 for ; Fri, 28 Feb 2003 15:41:45 -0500 (EST) Received: from www1.ietf.org (localhost.localdomain [127.0.0.1]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1SKoCp26857; Fri, 28 Feb 2003 15:50:12 -0500 Received: from ietf.org (odin.ietf.org [132.151.1.176]) by www1.ietf.org (8.11.6/8.11.6) with ESMTP id h1SKnip26822 for ; Fri, 28 Feb 2003 15:49:44 -0500 Received: from mx01.netapp.com (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA12591 for ; Fri, 28 Feb 2003 15:40:34 -0500 (EST) Received: from frejya.corp.netapp.com (frejya [10.10.20.91]) by mx01.netapp.com (8.12.6/8.12.6/NTAP-1.4) with ESMTP id h1SKgVgB027122 for ; Fri, 28 Feb 2003 12:42:32 -0800 (PST) Received: from tmt.netapp.com (tmt [10.33.25.87]) by frejya.corp.netapp.com (8.12.6/8.12.6/NTAP-1.4) with ESMTP id h1SKgVuk019112 for ; Fri, 28 Feb 2003 12:42:31 -0800 (PST) Message-Id: <5.2.0.9.2.20030228150337.02a5c820@silver.nane.netapp.com> X-Priority: 1 (Highest) Date: Fri, 28 Feb 2003 15:40:38 -0500 To: rddp@ietf.org From: "Talpey, Thomas" Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Subject: [rddp] Problem Statement and Architecture documents revised Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Two Working Group documents, the Problem Statement and also the Architecture, have been revised based on review and comments, in preparation for the March milestone of the RDDP WG charter. http://www.ietf.org/mail-archive/working-groups/rddp/current/msg00641.html Only minor changes to the references section of the Problem Statement were made. Changes to the Architecture were made as outlined below. Until the updated drafts appear in the internet-drafts directory, they are available at Tom. ----- Architecture document changes summary: The ddp_recv() and rdma_recv() did not make sufficiently clear that the data is received into a registered buffer. A new type, recv_message_t, was introduced and routines ddp_post_recv() and rdma_post_recv() were added. Text was added to clarify that the registration of a buffer on a socket_t does not preclude an implementation from supporting registration on a wider scope. Text was added to make it explicit that DDP does not provide framing, integrity and privacy services, and that these are a function of transport. A length specification was missing from the arguments of rdma_read(). RDMA Writes no longer result in an optional completion to the upper layer at the data sink. Some text and data types were changed for clarity, dates and references were updated, etc. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp