From exim@www1.ietf.org Fri Aug 1 09:56:42 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA20559 for ; Fri, 1 Aug 2003 09:56:42 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iaOF-0004O8-IA for rddp-archive@odin.ietf.org; Fri, 01 Aug 2003 09:56:16 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h71DuFEK016869 for rddp-archive@odin.ietf.org; Fri, 1 Aug 2003 09:56:15 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iaOF-0004O0-CN for rddp-web-archive@optimus.ietf.org; Fri, 01 Aug 2003 09:56:15 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA20473 for ; Fri, 1 Aug 2003 09:56:10 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iaOB-0002VX-00 for rddp-web-archive@ietf.org; Fri, 01 Aug 2003 09:56:11 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19iaO5-0002VK-00 for rddp-web-archive@ietf.org; Fri, 01 Aug 2003 09:56:05 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iaO3-0004Lj-8m; Fri, 01 Aug 2003 09:56:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iaNx-0004IK-FF for rddp@optimus.ietf.org; Fri, 01 Aug 2003 09:55:58 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id JAA20338 for ; Fri, 1 Aug 2003 09:55:50 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iaBt-0002FR-00 for rddp@ietf.org; Fri, 01 Aug 2003 09:43:29 -0400 Received: from mx01.netapp.com ([198.95.226.53]) by ietf-mx with esmtp (Exim 4.12) id 19iaBr-0002Ej-00 for rddp@ietf.org; Fri, 01 Aug 2003 09:43:27 -0400 Received: from hawk.corp.netapp.com (hawk [10.10.20.101]) by mx01.netapp.com (8.12.9/8.12.9/NTAP-1.5) with ESMTP id h71DgtQw021252; Fri, 1 Aug 2003 06:42:55 -0700 (PDT) Received: from svlexc01.hq.netapp.com (svlexc01.corp.netapp.com [10.10.22.171]) by hawk.corp.netapp.com (8.12.9/8.12.9/NTAP-1.5) with ESMTP id h71DggUv011629; Fri, 1 Aug 2003 06:42:55 -0700 (PDT) content-class: urn:content-classes:message MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable Subject: RE: [rddp] WG chair on buffer exhaustion (again) X-MimeOLE: Produced By Microsoft Exchange V6.0.6249.0 Date: Fri, 1 Aug 2003 06:42:45 -0700 Message-ID: Thread-Topic: [rddp] WG chair on buffer exhaustion (again) Thread-Index: AcNXx2uKah/k8YCqRAmHOR6xf4NtZQAatoRQ From: "Kanevsky, Arkady" To: "Mallikarjun C." , , Content-Transfer-Encoding: quoted-printable Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: quoted-printable Content-Transfer-Encoding: quoted-printable RC has has parts to its definition.=20 One of them states that any failure to deliver message (process posted WQE) over connection breaks the connection. Are you saying that failure to deliver message causes retransmission or that failure to deliver message causes CQE with ~success but connection stays? Arkady Kanevsky email: arkady@netapp.com Network Appliance phone: 781-768-5395 375 Totten Pond Rd. Fax: 781-895-1195 3rd Floor http: www.netapp.com Waltham, MA 02451-2010 general phone: 781-768-5300 > -----Original Message----- > From: Mallikarjun C. [mailto:cbm@rose.hp.com]=20 > Sent: Thursday, July 31, 2003 8:54 PM > To: Kanevsky, Arkady; Black_David@emc.com; rddp@ietf.org > Subject: Re: [rddp] WG chair on buffer exhaustion (again) >=20 >=20 > Arkady, >=20 > Please note the >=20 > "....MUST however ensure that the affected DDP Segment=20 > is eventually recovered ....way that guarantees ULP-transparency" >=20 > language in my suggested text. So the proposed text does not=20 > change anything wrt RDMAP and thus the automatic STag=20 > invalidation, and of course the definition of reliable=20 > connection stays the same as it was because the delivery=20 > semantics are still in-order. >=20 > Regards. > -- > Mallikarjun >=20 > Mallikarjun Chadalapaka > Networked Storage Architecture > Network Storage Solutions > Hewlett-Packard MS 5668=20 > Roseville CA 95747 > cbm@rose.hp.com >=20 > ----- Original Message -----=20 > From: "Kanevsky, Arkady" > To: ; > Sent: Thursday, July 31, 2003 8:37 AM > Subject: RE: [rddp] WG chair on buffer exhaustion (again) >=20 >=20 > > David, > > I am trying to comprehand the 2 issues. > >=20 > > You stated that Mallikarjun's issue involves statistical (under-) > > provisioning of buffers can happen without any Shared > > Receive queues. > > 1. What is the definition of reliable connection based on=20 > this change? > > 2. How does UPL knows/control what is the connection behavior: > > connection broken(termination DDP stream) if no recv buffer is > > availble vs. > > Send message dropped on the floor and not delivered, but DDP > > connection stays? > > 3. How is the Stag invalidation effected (if at all) with=20 > both proposals? > >=20 > > Thanks, > > Arkady > >=20 > > Arkady Kanevsky email:=20 > arkady@netapp.com > > Network Appliance phone: 781-768-5395 > > 375 Totten Pond Rd. Fax: 781-895-1195 > > 3rd Floor =20 > http: www.netapp.com > > Waltham, MA 02451-2010 general phone: 781-768-5300 > >=20 > >=20 > > > -----Original Message----- > > > From: Black_David@emc.com [mailto:Black_David@emc.com] > > > Sent: Wednesday, July 30, 2003 5:40 PM > > > To: rddp@ietf.org > > > Subject: [rddp] WG chair on buffer exhaustion (again) > > >=20 > > >=20 > > > With WG Chair hat on, let me step into this debate: > > >=20 > > > > > Section 3.2 to: > > > > > " Note, however, that the ULP needs to address=20 > flow control > > > > > issues. > > > Any > > > > > ULP that runs on DDP SHOULD consider the lack of an > > > associated > > > receive > > > > > buffer to place an inbound Untagged Message of=20 > the ULP as=20 > > > > > an > > > exception." > > > >=20 > > > > If the local interface has allows the DDP endpoint to know > > > the valid > > > > range of MSNs on a stream (which it will for simple Receive > > > Queues at > > > > the minimum) then it definitely SHOULD NOT consider the=20 > lack of a > > > > buffer for an *invalid message* to be "an exception". > > >=20 > > > Two issues are getting conflated here, and need to be=20 > kept separate: > > > - Mallikarjun's issue involves statistical (under-) > > > provisioning of buffers, > > > so that an otherwise valid untagged message shows up=20 > > > without a buffer > > > available to receive it (if the implementation had done=20 > > > worst-case > > > provisioning, there would have been a buffer=20 > > > available). I believe > > > there is WG rough consensus that this is an exceptional=20 > > > condition > > > that does not need to immediately cause the termination=20 > > > of the DDP > > > stream that sent the message, but it SHOULD not occur=20 > > > frequently. > > > Mallikarjun's proposed text is close enough to what's=20 > > > needed for this. > > > This condition could happen even on a simple Receive Queue if > > > statistical (under-) provisioning is being used. > > > - Caitlin's issue involves a peer violating a fixed=20 > > > per-DDP-stream limit on > > > the number of untagged messages allowed . At the=20 > > > moment, the exact > > > need and nature of such limits is still an open issue=20 > > > that needs to be > > > resolved in the context of the security draft. When we=20 > > > have it resolved > > > there, additional text will (probably) get driven into=20 > > > the DDP draft. > > > If the issue is resolved in the manner Caitlin advocates, then > > > her proposed text is certainly a candidate for what the=20 > > > DDP draft > > > will need to say, but at the moment, the issue is open, and the > > > DDP draft should not change what it says about this issue until > > > it is resolved in the context of the security draft. > > >=20 > > > Also, I've seen some confusing references to iSER and > > > assumptions about local interface and resource sharing model.=20 > > > Shared Receive Queues have > > > *not* been approved/adopted by this WG (and can't be until we=20 > > > get the resource sharing issue [2nd bullet above] resolved). =20 > > > To the extent that iSER requires Shared Receive Queues and=20 > > > can't cope with any other local resource model, it's probably=20 > > > made an overly restrictive design assumption that it's=20 > > > designers would be well-advised to reconsider, keeping in=20 > > > mind that the Hilland verbs draft is at best an *example* of=20 > > > a local interface to RDDP; it should not be viewed as=20 > > > normative functional requirements on that local interface. > > >=20 > > > With luck, that's useful clarification, but feel free to ask > > > if anything above doesn't make sense ... > > >=20 > > > Thanks, > > > --David > > > ---------------------------------------------------- > > > David L. Black, Senior Technologist > > > EMC Corporation, 176 South St., Hopkinton, MA 01748 > > > +1 (508) 293-7953 FAX: +1 (508) 293-7786 > > > black_david@emc.com Mobile: +1 (978) 394-7754 > > > ---------------------------------------------------- > > >=20 > > > _______________________________________________ > > > rddp mailing list > > > rddp@ietf.org > > > https://www1.ietf.org/mailman/listinfo/rddp > > >=20 > >=20 > > _______________________________________________ > > rddp mailing list > > rddp@ietf.org > > https://www1.ietf.org/mailman/listinfo/rddp > >=20 >=20 >=20 > _______________________________________________ > rddp mailing list > rddp@ietf.org > https://www1.ietf.org/mailman/listinfo/rddp >=20 _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Aug 1 10:52:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA25587 for ; Fri, 1 Aug 2003 10:52:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ibGG-0006eF-FI for rddp-archive@odin.ietf.org; Fri, 01 Aug 2003 10:52:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h71Eq4jV025551 for rddp-archive@odin.ietf.org; Fri, 1 Aug 2003 10:52:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ibGG-0006e2-CO for rddp-web-archive@optimus.ietf.org; Fri, 01 Aug 2003 10:52:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA25583 for ; Fri, 1 Aug 2003 10:51:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ibGD-0003A5-00 for rddp-web-archive@ietf.org; Fri, 01 Aug 2003 10:52:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ibGD-0003A2-00 for rddp-web-archive@ietf.org; Fri, 01 Aug 2003 10:52:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ibGD-0006dP-DJ; Fri, 01 Aug 2003 10:52:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ibGA-0006dB-0N for rddp@optimus.ietf.org; Fri, 01 Aug 2003 10:51:58 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id KAA25574 for ; Fri, 1 Aug 2003 10:51:52 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ibG5-00039n-00 for rddp@ietf.org; Fri, 01 Aug 2003 10:51:53 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19ibG5-00039j-00 for rddp@ietf.org; Fri, 01 Aug 2003 10:51:53 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 05CA1245312; Fri, 1 Aug 2003 10:51:31 -0400 (EDT) Date: Fri, 1 Aug 2003 09:51:18 -0500 Subject: Re: [rddp] WG chair on buffer exhaustion (again) Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: "Mallikarjun C." , , To: "Kanevsky, Arkady" From: Caitlin Bestler In-Reply-To: Message-Id: <9B808C58-C42F-11D7-A551-003065D48EE0@asomi.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Friday, August 1, 2003, at 08:42 AM, Kanevsky, Arkady wrote: > RC has has parts to its definition. > One of them states that any failure to deliver message (process posted > WQE) over connection breaks the connection. > > Are you saying that failure to deliver message causes retransmission or > that failure to deliver message causes CQE with ~success but connection > stays? Any failure will result in terminating the stream (which for MPA/TCP means breaking the connection). The proposal is to essentially allow a "conditional placement" interface from the LLP to DDP. If a segment/chunk would not be placeable immediately, the LLP would have the *option* of *delaying* placement. Exactly what tricks it would use to delay placement are out of scope, other than there is an expressed preference that forcing the other side to do an LLP retransmit is NOT preferred and should be used minimally and only as a last resort. If the LLP wants to volunteer to hold the buffers for awhile, that is just fine with DDP. The LLP merely *wants* to minimize its own buffering. But it MAY choose to provide that service. DDP, by contrast, does not buffer. So whatever delay mechanism is used, LLP buffering or LLP retransmission, there is simply no ULP visible event to be placed on the CQE until placement *and* delivery is successful or the LLP insists that DDP accept placement *now* and DDP is forced to terminate the stream for lack of a buffer. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Aug 1 11:19:28 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA26155 for ; Fri, 1 Aug 2003 11:19:28 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ibgM-0007IM-3d for rddp-archive@odin.ietf.org; Fri, 01 Aug 2003 11:19:02 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h71FJ2tf028036 for rddp-archive@odin.ietf.org; Fri, 1 Aug 2003 11:19:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ibgM-0007I7-0J for rddp-web-archive@optimus.ietf.org; Fri, 01 Aug 2003 11:19:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA26149 for ; Fri, 1 Aug 2003 11:18:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ibgL-0003Iu-00 for rddp-web-archive@ietf.org; Fri, 01 Aug 2003 11:19:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19ibgK-0003Ir-00 for rddp-web-archive@ietf.org; Fri, 01 Aug 2003 11:19:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ibgK-0007Hd-LR; Fri, 01 Aug 2003 11:19:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ibg7-0007H7-1y for rddp@optimus.ietf.org; Fri, 01 Aug 2003 11:18:47 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA26143 for ; Fri, 1 Aug 2003 11:18:43 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ibg6-0003Ie-00 for rddp@ietf.org; Fri, 01 Aug 2003 11:18:46 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19ibg5-0003IU-00 for rddp@ietf.org; Fri, 01 Aug 2003 11:18:45 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 784F1245F45; Fri, 1 Aug 2003 11:18:25 -0400 (EDT) Date: Fri, 1 Aug 2003 10:18:12 -0500 Subject: Re: [rddp] Re: iSCSI/iWARP drafts and flow control Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: , To: "Mallikarjun C." From: Caitlin Bestler In-Reply-To: <020c01c357c6$0a21e340$18a4f40f@rose.hp.com> Message-Id: <5DA68BAA-C433-11D7-A551-003065D48EE0@asomi.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Thursday, July 31, 2003, at 07:43 PM, Mallikarjun C. wrote: >> No new wire protocol is required. > > Please explain to me how credit can be replenished > in the following example - without new positive flow > control protocol. I thought of at least one algorithm overnight. This is not presented as a definitive proposal, merely to show a strategy that can allow iSER to provide flow control of *all* untagged messages. iSER untagged messages are divided into a predictable portion regulated by CmdSNs in a very direct fashion and asynchronous messages. The latter category can be characterized as having a low sustained rate compared to the CmdSN-related untagged messages, but that the traffic is very bursty. That is, the peak rate can be high. That suggests adapting a classic "leaky bucket" credit system of the type typically used to regulate burst traffic over rate controlled networks. The key difference is that the "clock" is the Max CmdSN. What is required is the following: The sender maintains a asynch-credit counter. It is initialized to a known value. This value could be negotiated if it is believed that there is enough variation to warrant negotiation, otherwise it would be fixed. That caveat applies to all other "constants" cited in this algorithm. When the ULP desires to send an untagged message using one of these credits: The credit count is brought up to date: The current Max CmdSN is compared with the value when the prior "fringe" send was performed. The delta is used to grant new credits, however there is a maximum number of credits that may be accumulated. If there are enough credits: the credit count is reduced and the untagged message may be sent. If there are not enough credits: the untagged message must be delayed until the Max CmdSN is advanced. After a configurable delay, iSER SHOULD send some form of NOP command to cause Max CmdSN to advance. Note that there are NO iSER defined messages for the purpose of flow control. Untagged messages may be delayed, but that could happen over TCP just as easily. The only change is that *only* untagged messages will be delayed, never tagged messages, and *where* they are delayed (iSER versus TCP). _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Aug 1 17:15:00 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA12729 for ; Fri, 1 Aug 2003 17:14:58 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ihEO-0007aK-Ew for rddp-archive@odin.ietf.org; Fri, 01 Aug 2003 17:14:32 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h71LEWUt029152 for rddp-archive@odin.ietf.org; Fri, 1 Aug 2003 17:14:32 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ihEO-0007a7-Bt for rddp-web-archive@optimus.ietf.org; Fri, 01 Aug 2003 17:14:32 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA12721 for ; Fri, 1 Aug 2003 17:14:27 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ihDt-0007Xy-Gf; Fri, 01 Aug 2003 17:14:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19ihDi-0007Xj-Ho for rddp@optimus.ietf.org; Fri, 01 Aug 2003 17:13:50 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA12716 for ; Fri, 1 Aug 2003 17:13:45 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19ihDg-0006yB-00 for rddp@ietf.org; Fri, 01 Aug 2003 17:13:48 -0400 Received: from [168.159.2.31] (helo=mailhub.lss.emc.com) by ietf-mx with esmtp (Exim 4.12) id 19ihDf-0006y8-00 for rddp@ietf.org; Fri, 01 Aug 2003 17:13:47 -0400 Received: from mxic1.corp.emc.com (mxic1.isus.emc.com [168.159.129.100]) by mailhub.lss.emc.com (Switch-2.2.5/Switch-2.2.0) with ESMTP id h71LDhS06316; Fri, 1 Aug 2003 17:13:44 -0400 (EDT) Received: by mxic1.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Fri, 1 Aug 2003 17:13:43 -0400 Message-ID: <277DD60FB639D511AC0400B0D068B71E0C2467CB@corpmx14.corp.emc.com> To: Arkady.Kanevsky@netapp.com, Black_David@emc.com, rddp@ietf.org Subject: RE: [rddp] WG chair on buffer exhaustion (again) Date: Fri, 1 Aug 2003 17:02:51 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , > David, > I am trying to comprehand the 2 issues. > > You stated that Mallikarjun's issue involves statistical (under-) > provisioning of buffers can happen without any Shared Receive queues. Suppose the ULP allows its one peer to deliver 70 messages but only provisions 25 local buffers. While not all of the drafts allow this, I would expect to see it done in practice. > 1. What is the definition of reliable connection based on this change? Unchanged. Any failure to deliver a DDP message to an application terminates the DDP stream. In this case "failure" means "will not be delivered, ever". The discussion around Mallikarjun's issue involves allowing implementations to do things that convert what would have been a "failure" (e.g., no ULP buffer available) to a "delay" (e.g., copy to a kernel buffer, wait for the ULP to provide a ULP buffer) in rare circumstances. > 2. How does UPL knows/control what is the connection behavior: > connection broken(termination DDP stream) if no recv buffer is availble vs. > Send message dropped on the floor and not delivered, but DDP connection stays? Latter case is not possible - if the implementation drops the Send on the floor so that the ULP never sees it, the DDP stream on which this occurred is terminated. > 3. How is the Stag invalidation effected (if at all) with both proposals? No effect, it's an orthogonal issue (but one that will have to be taken up). Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Sat Aug 2 11:28:32 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA15984 for ; Sat, 2 Aug 2003 11:28:32 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iyIi-0004P2-79 for rddp-archive@odin.ietf.org; Sat, 02 Aug 2003 11:28:08 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h72FS8GP016920 for rddp-archive@odin.ietf.org; Sat, 2 Aug 2003 11:28:08 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iyIi-0004Op-2u for rddp-web-archive@optimus.ietf.org; Sat, 02 Aug 2003 11:28:08 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA15976 for ; Sat, 2 Aug 2003 11:28:02 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iyIh-0003pr-00 for rddp-web-archive@ietf.org; Sat, 02 Aug 2003 11:28:07 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19iyIg-0003pn-00 for rddp-web-archive@ietf.org; Sat, 02 Aug 2003 11:28:06 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iyIa-0004OS-Km; Sat, 02 Aug 2003 11:28:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iyIL-0004OF-Ae for rddp@optimus.ietf.org; Sat, 02 Aug 2003 11:27:45 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA15971 for ; Sat, 2 Aug 2003 11:27:39 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iyIK-0003pk-00 for rddp@ietf.org; Sat, 02 Aug 2003 11:27:44 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19iyIJ-0003pg-00 for rddp@ietf.org; Sat, 02 Aug 2003 11:27:43 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 36C38244F39 for ; Sat, 2 Aug 2003 11:27:23 -0400 (EDT) Date: Sat, 2 Aug 2003 10:27:10 -0500 Mime-Version: 1.0 (Apple Message framework v552) Content-Type: text/plain; charset=US-ASCII; format=flowed From: Caitlin Bestler To: rddp@ietf.org Content-Transfer-Encoding: 7bit Message-Id: X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Subject: [rddp] RDMA Read - Data Sink STag Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit In section 6.4, the RDMAP specification specifies the following about the Data Sink Steering Tag: > The Data Sink Steering Tag identifies the Data > Sink's Tagged Buffer. This field MUST be copied, > without interpretation, from the RDMA Read > Request into the corresponding RDMA Read Response > and allows the Data Sink to place the returning > data. The STag is associated with the RDMAP > Stream through a mechanism that is outside the > scope of the RDMAP specification (see Section > 10.3 Other Security Considerations). The intent of this clause is to ensure that a) RDMA Read Replies obtain their STag strictly from the RDMA Read Request, without interpretation. b) The RDMA Read Request cannot be inferred to be a general form of "buffer advertisement". That is, it was intended to be compatible with the STag being a completely normal STag that just happens to be in use for RDMA Reads, and for it to be a special form of STag that is limited in scope to supporting RDMA Reads only on a specific DDP Stream. One strategy for multi-client servers is to avoid advertising their own buffers. Clients advertise their buffers. The server RDMA Writes to them, or RDMA Reads from them. In such an environment it would be a highly desirable feature of the local interface to restrict the usage of an STag to RDMA Read responses on a single DDP Stream. And further, to automatically limit its lifespan to that of the RDMA Read Work Request. Essentially, the "RDMA Read only" STag would be a reference to a pending local request to perform an RDMA Read. There is one minor problem point in the draft that is inconsistent with allowing an implementation to use this strategy. The protocol overview section (3.2) says the following in clause 6: > The Data Sink STag associated with the Tagged Buffer > remains valid until the ULP at the Data Sink invalidates > it. If there is no objection, I would like to have that changed to: The Data Sink STag associated with the Tagged Buffer remains valid until it is invalidated at the direction of the Data Sink ULP. This makes it clear that an implementation may have a local interface that, with permission of the ULP, automatically invalidates an STag at completion of an RDMA Read operation. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Sat Aug 2 11:57:27 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA16206 for ; Sat, 2 Aug 2003 11:57:27 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iykh-0006Tt-1M for rddp-archive@odin.ietf.org; Sat, 02 Aug 2003 11:57:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h72Fv3nQ024907 for rddp-archive@odin.ietf.org; Sat, 2 Aug 2003 11:57:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iykg-0006Te-TR for rddp-web-archive@optimus.ietf.org; Sat, 02 Aug 2003 11:57:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA16190 for ; Sat, 2 Aug 2003 11:56:56 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iykf-0003u3-00 for rddp-web-archive@ietf.org; Sat, 02 Aug 2003 11:57:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19iykf-0003u0-00 for rddp-web-archive@ietf.org; Sat, 02 Aug 2003 11:57:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iykf-0006Sb-6e; Sat, 02 Aug 2003 11:57:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19iykX-0006LG-DR for rddp@optimus.ietf.org; Sat, 02 Aug 2003 11:56:53 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA16187 for ; Sat, 2 Aug 2003 11:56:47 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19iykW-0003tw-00 for rddp@ietf.org; Sat, 02 Aug 2003 11:56:52 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19iykV-0003tt-00 for rddp@ietf.org; Sat, 02 Aug 2003 11:56:51 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 94CA72463A8 for ; Sat, 2 Aug 2003 11:56:31 -0400 (EDT) Date: Sat, 2 Aug 2003 10:56:18 -0500 Mime-Version: 1.0 (Apple Message framework v552) Content-Type: text/plain; charset=US-ASCII; format=flowed From: Caitlin Bestler To: rddp@ietf.org Content-Transfer-Encoding: 7bit Message-Id: X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Subject: [rddp] Security Concern: RDMA Read Data Sink STag Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit There is an additional security vulnerability dealing with RDMA Reads that should be added to the security draft. This one deals with deliberate abuse on the part of the remote peer. A common security practice for servers that deal with multiple clients from a single daemon will be to simply refrain from advertising server buffers. The DAFS protocol is one example of this strategy. However, the strategy is likely to be common. Advertising a tagged buffer is a resource commitment. Making precise resource commitments to many transient clients in parallel will tie up any server's resources quickly. Under this strategy, the server instead accepts advertisements of tagged buffers in each client request (delivered as an untagged message). The server then performs the necessary data transfers to/from the client's buffers before sending the response via a RDMA Send with Invalidate. Because the daemon is supporting many clients, and because it is not advertising buffers, it is likely that it will create only a single Protection Domain. Under most RDMA-enabled local interfaces, the Data Sink STag is provided by the consumer in the work request for the RDMA Read. It is a general purpose STag. This is not a vulnerability to any form of accidental or benign attack. The Data Sink STag given in a RDMA Read Request should never be exposed to the peer ULP. It should be seen no higher than the Data Source RDMAP layer. A deliberate attacker, however, could eavesdrop on traffic to determine the value of a currently exposed RDMA Read STag. That STag could then be used on a different DDP Stream. Neither the RDMAP or DDP drafts require an RDMA Read Data Sink STag be valid for any purpose other than a single RDMA Read Reply. However, neither requires that the STag be invalid for other purposes. Indeed, clearly it is acceptable for the local interface to place responsibility for managing the scope and duration of Data Sink STags on the ULP. The ULP needs to consider the scope of the Data Sink STag that will be used in RDMA Reads when RDMA Reads will be used in an environment where hostile attacks are possible. Options offered by the local interface to limit the scope and/or duration of the STag should be employed as appropriate. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Wed Aug 6 19:27:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA09983 for ; Wed, 6 Aug 2003 19:27:28 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19kXgM-0007WG-T0 for rddp-archive@odin.ietf.org; Wed, 06 Aug 2003 19:27:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h76NR2k4028900 for rddp-archive@odin.ietf.org; Wed, 6 Aug 2003 19:27:02 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19kXgM-0007W3-Or for rddp-web-archive@optimus.ietf.org; Wed, 06 Aug 2003 19:27:02 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA09971 for ; Wed, 6 Aug 2003 19:26:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19kXgL-0000OL-00 for rddp-web-archive@ietf.org; Wed, 06 Aug 2003 19:27:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19kXgK-0000OH-00 for rddp-web-archive@ietf.org; Wed, 06 Aug 2003 19:27:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19kXgK-0007Un-AO; Wed, 06 Aug 2003 19:27:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19kXfk-0007UV-LJ for rddp@optimus.ietf.org; Wed, 06 Aug 2003 19:26:24 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id TAA09956; Wed, 6 Aug 2003 19:26:19 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19kXfi-0000Nq-00; Wed, 06 Aug 2003 19:26:22 -0400 Received: from maho3msx2.isus.emc.com ([128.221.11.32] helo=MAHO3MSX2.corp.emc.com) by ietf-mx with esmtp (Exim 4.12) id 19kXfi-0000Nf-00; Wed, 06 Aug 2003 19:26:22 -0400 Received: by maho3msx2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Wed, 6 Aug 2003 19:25:52 -0400 Message-ID: To: minutes@ietf.org, rddp@ietf.org Date: Wed, 6 Aug 2003 19:25:51 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain Subject: [rddp] RDDP WG Vienna minutes Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , RDDP WG minutes - Vienna, Austria, July 2003 -------------------------------------------- Thanks to Tom Talpey for taking the initial minutes. Letters in [square brackets] indicate first letter in name of corresponding presentation in proceedings. Tuesday, July 15 5-6pm ---------------------- David Black (WG chair) introduces agenda changes, no bashing, agenda accepted [A] Review of WG draft intentions and status - see presentation [B] + RDDP Applicability statement, Lode Coene [C] Review of document content - draft considered by authors to be in good shape, waiting for comments from WG. David Black requests that TCP mapping issues (drawbacks of TCP by comparison to SCTP) be included in the applicability statement. + SCTP Mapping draft [D] On its third revision. A DDP Stream establishment issue has been identified, can a protection domain assignment be deferred or no? Needs documentation at a minimum. A draft revision under author review concerns single-message exchange for "private data", compatible with current MPA draft, but this is part of the open startup issues in the MPA draft. No others issues are currently identified, speak up now... New version expected in next few weeks. + Shared receive queues vs Shared buffer pool [D] David Black presented this issue from the mailing list on behalf of Caitlin Bestler, who was unable to travel to Vienna. Shared receive queues are proposed in draft-hilland-verbs-00. Their presence/absence has impact on other drafts: Architecture document should it define a "post receive"? Security issues The problem: need to underprovision receive buffers, but how to control per-receive queue quotas? Two proposals: Shared receive queue (draft-hilland) Shared buffer pool (email on list) The proposals are described and a lengthy discussion ensues. David Black suggests trying the following tack: need to resolve robustness to DOS attack on shared buffers. Tag this as the only open portion of this issue and look at countermeasures in security draft. Examine offline and continue this issue on Thursday. A question is asked about why this issue is holding up the architecture draft, which is *not* an interface specification. David Black answers that the architecture draft goes into sufficient detail on pseudo-interfaces that some discussion of resource sharing is necessary there. + RDMAP/DDP Security draft - Jim Pinkerton [E] Presents the new security draft for first time. Draft is focused on wire protocol issues, in a generic way with out referencing verbs, etc. Sketches components necessary to implement security over RDMA. - privileged resource manager, controlling the RNIC itself as well as two classes of application - priv and non-priv applications. Identifies resources to be analyzed (and controlled): connection context, data buffers, translation tables, STags, completion queues, plus RDMA Read request queue. Describes the "dimensions of trust": local resource sharing, local trust and remote trust. There are 8 combinations of trust, 4 are addressed in the paper. The 4 were chosen as most applicable to IETF areas of interest, due to brevity and also lack of relevant applications which might fall into the other 4. Enumerates tools for countermeasures: protection domain, limiting STag scope, (4 others) David Black requests that for Thursday's session, protocols should decide which "trust bucket" each lands in, determines if bucket is addressed in draft, and if so are the issues accurate? Thursday, July 17, 3:30-5:30pm ------------------------------ + Architecture draft resolution ddp_post_receive goes back in Allow sharing of buffers for untagged receives Work out how to avoid tacking limit per stream/socket problem in arch draft; problem will be addressed in security draft In ddp draft, ok to not fail immediately on shared overdraw question from floor - have we addressed the issues raised by VJ delay-bandwidth results? Enough buffers to allow the flow to use all available bw? A (David Black): The fact that RDDP moves control of buffering resources upwards in the protocol stack doesn't change the need to provide bandwidth-delay product quantity of buffering to enable full use of the available network bandwidth. This is an issue for receivers to appropriately provision receive buffers. Consider adding a statement to this effect to the architecture draft. + Return to Security document discussion, Jim Pinkerton. [This meeting time conflicts with SAAG meeting so security experts may not be in attendance.] Current open issues summarized: - IPSec section is currently a placeholder, and will be filled in (Bernard Aboba to help out) - Also, summary section at end still in progress. - From reflector, elaborate some issues in "trust model". - Multiple Protection Domains - Scope of STags - Other resources to add to security discussion (async event queue, PD, etc) - STag disable/enable by non-privileged app. Question from floor - we need an analysis of issues introduced by RDMA into existing protocols, e.g. NFSv4. When transfers are done purely by RDMA they may not be protected by existing security. A (David Black): This may indeed require additional security coverage such as IPsec which may be enabled by existing CCM NFSv4 proposal. It as also noted that the integrity protection is present in NFSv4, RPCSEC_GSS integrity, can verify that RDMA transfers actually occur as promised, while IPsec can be used to enforce the authorizations themselves. Two different issues, important to keep separate. Question - should performance analysis be included in our analysis? Especially where IPSec overhead is involved. A (David Black): First of all, let's complete the security analysis. At that point, the specific performance implications may be worth analyzing. But not before. Next major revision of this document should become an official WG draft. No objections. + Status of TCP mapping issues WG chair summarizes marker issues resolved to date: Sender alignment Marker usage, intervals, optionality CRC use (5+ issues) Further discussion to go into applicability draft Question from floor - have we worked out when it's ok not to use MPA CRC yet? Prefers to have it always present for simplicity. A (David Black): Rough consensus is for CRC to be optional. Discussion of when it may be disabled and what ULP designers/implementers should assume goes into the applicability statement draft. Last(?) CRC Issue - one CRC or two? Case for two - faster placement, fast header validation Case for one - tractable, simpler, doesn't abuse layering Statement from floor - there are more reasons to have two CRCs: DDP Routers don't have to rebuild data crc Don't have to wait for whole segment to begin processing header Large amount of controversy whether it's valid to rewrite DDP headers. WG Chair declares that rough consensus exists for one CRC; while there are advantages to two, they don't outweigh increased complexity. Further discussion will be on the list. + MPA discussion - Paul Culley Changes to most recent draft recapped: optional, receiver-specified markers optional, mutually agreed CRC MPA streaming mode "key" validation mechanism at connect new informational sections Authors' additional proposals no "alignment" of market to TCP sequence #, as alignment would violate layering CRC field always present, not filed in or checked when not used - simplicity no private data in MPA (carried only in TCP mode), quiescing stream required at MPA init MPA 128-bit start key, carrying a fixed MPA signal + CRC/marker bits, etc. passive/active rules specified Question - what happens if both "active"? Needs to be addressed Resolved - Start Key should at a minimum assert Active vs. Passive. Discussion of connection startup models - immediate RDMA enabled, RDMA enabled midstream, RDMA midstream with private data (blob follows MPA key). This whole area of how to initiate a connection, including private data, is open and needs to go to list Chair proposes that this document is ready to go to official WG draft for TCP mapping of RDDP (with significant issues to be worked out of course). No objections. Question: Is the MPA draft as informational or standards track? A (WG Chair) Informational: per ADs from recent rechartering. Can be reconsidered when draft is complete and ready for publication as an RFC. Q: Didn't acceptance of Start Key mechanism enable standards track status? A (WG Chair): No, refusal to accept Start Key mechanism would have resulted in TCP mapping work being removed from RDDP WG charter. NOTE: WG Chair has subsequently verified w/ADs that standards track status for the TCP mapping (MPA) can be reconsidered when the draft is complete. Among the reasons for this is a strong desire to see exactly what is being proposed and understand its effects on TCP before approving standards track status. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Aug 7 13:59:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA24399 for ; Thu, 7 Aug 2003 13:59:28 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19kp2V-0004Wy-Om for rddp-archive@odin.ietf.org; Thu, 07 Aug 2003 13:59:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h77Hx3fm017412 for rddp-archive@odin.ietf.org; Thu, 7 Aug 2003 13:59:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19kp2V-0004Wl-LJ for rddp-web-archive@optimus.ietf.org; Thu, 07 Aug 2003 13:59:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA24391 for ; Thu, 7 Aug 2003 13:58:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19kp2T-0001Eb-00 for rddp-web-archive@ietf.org; Thu, 07 Aug 2003 13:59:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19kp2S-0001EY-00 for rddp-web-archive@ietf.org; Thu, 07 Aug 2003 13:59:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19kp2T-0004WM-FX; Thu, 07 Aug 2003 13:59:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19koxO-00048E-CX for rddp@optimus.ietf.org; Thu, 07 Aug 2003 13:53:46 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id NAA24211 for ; Thu, 7 Aug 2003 13:53:40 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19koxM-0001AI-00 for rddp@ietf.org; Thu, 07 Aug 2003 13:53:44 -0400 Received: from e31.co.us.ibm.com ([32.97.110.129]) by ietf-mx with esmtp (Exim 4.12) id 19koxK-00019m-00 for rddp@ietf.org; Thu, 07 Aug 2003 13:53:42 -0400 Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e31.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h77HqoRZ263792; Thu, 7 Aug 2003 13:52:50 -0400 Received: from d03nm694.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h77HqnsE102452; Thu, 7 Aug 2003 11:52:49 -0600 Importance: Normal MIME-Version: 1.0 To: Caitlin Bestler Cc: , X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: Mike Ko Date: Thu, 7 Aug 2003 10:52:46 -0700 X-MIMETrack: Serialize by Router on D03NM694/03/M/IBM(Release 6.0.1 [IBM]|June 10, 2003) at 08/07/2003 11:52:49, Serialize complete at 08/07/2003 11:52:49 Content-Type: text/plain; charset="us-ascii" Subject: [rddp] Re: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Caitlin, since MaxCmdSN is used by the target to limit the number of commands it can receive from the initiator, your suggestion would work for limiting the number of "fringe" messages the initiator can send to the target. However, for "fringe" messages from the target to the initiator, such as the Aysnchronous Message, the target cannot simply raise MaxCmdSN and expect the initiator to increase the number of receive buffers for "fringe" messages correspondingly. Mallikarjun is correct in pointing out that some form of new wire protocol for flow control is required if we need to flow control the messages not regulated by CmdSN. Mike Ko IBM Almaden Research mako@almaden.ibm.com Sent by: owner-ips@ece.cmu.edu To: "Mallikarjun C." cc: , Subject: Re: [rddp] Re: iSCSI/iWARP drafts and flow control On Thursday, July 31, 2003, at 07:43 PM, Mallikarjun C. wrote: >> No new wire protocol is required. > > Please explain to me how credit can be replenished > in the following example - without new positive flow > control protocol. I thought of at least one algorithm overnight. This is not presented as a definitive proposal, merely to show a strategy that can allow iSER to provide flow control of *all* untagged messages. iSER untagged messages are divided into a predictable portion regulated by CmdSNs in a very direct fashion and asynchronous messages. The latter category can be characterized as having a low sustained rate compared to the CmdSN-related untagged messages, but that the traffic is very bursty. That is, the peak rate can be high. That suggests adapting a classic "leaky bucket" credit system of the type typically used to regulate burst traffic over rate controlled networks. The key difference is that the "clock" is the Max CmdSN. What is required is the following: The sender maintains a asynch-credit counter. It is initialized to a known value. This value could be negotiated if it is believed that there is enough variation to warrant negotiation, otherwise it would be fixed. That caveat applies to all other "constants" cited in this algorithm. When the ULP desires to send an untagged message using one of these credits: The credit count is brought up to date: The current Max CmdSN is compared with the value when the prior "fringe" send was performed. The delta is used to grant new credits, however there is a maximum number of credits that may be accumulated. If there are enough credits: the credit count is reduced and the untagged message may be sent. If there are not enough credits: the untagged message must be delayed until the Max CmdSN is advanced. After a configurable delay, iSER SHOULD send some form of NOP command to cause Max CmdSN to advance. Note that there are NO iSER defined messages for the purpose of flow control. Untagged messages may be delayed, but that could happen over TCP just as easily. The only change is that *only* untagged messages will be delayed, never tagged messages, and *where* they are delayed (iSER versus TCP). _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Aug 7 15:44:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA29810 for ; Thu, 7 Aug 2003 15:44:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19kqg8-000231-8q for rddp-archive@odin.ietf.org; Thu, 07 Aug 2003 15:44:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h77Ji4FD007867 for rddp-archive@odin.ietf.org; Thu, 7 Aug 2003 15:44:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19kqg8-00022o-5Q for rddp-web-archive@optimus.ietf.org; Thu, 07 Aug 2003 15:44:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA29801 for ; Thu, 7 Aug 2003 15:43:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19kqg6-0002Em-00 for rddp-web-archive@ietf.org; Thu, 07 Aug 2003 15:44:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19kqg6-0002Ej-00 for rddp-web-archive@ietf.org; Thu, 07 Aug 2003 15:44:02 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19kqg5-00022J-45; Thu, 07 Aug 2003 15:44:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19kqfU-000220-Bj for rddp@optimus.ietf.org; Thu, 07 Aug 2003 15:43:24 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id PAA29790 for ; Thu, 7 Aug 2003 15:43:19 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19kqfS-0002ET-00 for rddp@ietf.org; Thu, 07 Aug 2003 15:43:22 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19kqfS-0002EP-00 for rddp@ietf.org; Thu, 07 Aug 2003 15:43:22 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id ADB8024522C; Thu, 7 Aug 2003 15:43:04 -0400 (EDT) Date: Thu, 7 Aug 2003 14:42:46 -0500 Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: , To: Mike Ko From: Caitlin Bestler In-Reply-To: Message-Id: <51FDA22D-C90F-11D7-ABA1-003065D48EE0@asomi.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Subject: [rddp] Re: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Thursday, August 7, 2003, at 12:52 PM, Mike Ko wrote: > Caitlin, since MaxCmdSN is used by the target to limit the number of > commands it can receive from the initiator, your suggestion would work > for > limiting the number of "fringe" messages the initiator can send to the > target. However, for "fringe" messages from the target to the > initiator, > such as the Aysnchronous Message, the target cannot simply raise > MaxCmdSN > and expect the initiator to increase the number of receive buffers for > "fringe" messages correspondingly. > > Mallikarjun is correct in pointing out that some form of new wire > protocol > for flow control is required if we need to flow control the messages > not > regulated by CmdSN. To be more precise, no general purpose wire protocol is required that must act as a filter on every untagged message. The procedure I outlined deals with "fringe" messages from the initiator. It is logic that is never needed more often than per-fringe-message, and generally requires no additional messages. If fail to see why that cannot be worked out for each category of message that iSCSI requires. All that is required is to identify some acknowledgment that flows in the desired direction. Under the worst case you might be forced to create a message for the sole purpose of restoring credits. You cannot claim that SCSI functionality over RDMA is incompatible with flow control. SCSI over TCP/IP is flow controlled by TCP, iSER flow control would be *less* restrictive than TCP flow control, but it does have to be identified and specified. Additionally SRP is fully defined to work over RDMA with flow control. The problem has a solution. There is no justification for exempting iSER from the responsibility to provide ULP flow control. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Thu Aug 7 21:29:27 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA11335 for ; Thu, 7 Aug 2003 21:29:27 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19kw3z-0006nN-9L for rddp-archive@odin.ietf.org; Thu, 07 Aug 2003 21:29:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h781T3X9026117 for rddp-archive@odin.ietf.org; Thu, 7 Aug 2003 21:29:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19kw3z-0006nA-6I for rddp-web-archive@optimus.ietf.org; Thu, 07 Aug 2003 21:29:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA11323 for ; Thu, 7 Aug 2003 21:28:56 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19kw3w-0004M2-00 for rddp-web-archive@ietf.org; Thu, 07 Aug 2003 21:29:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19kw3v-0004Ly-00 for rddp-web-archive@ietf.org; Thu, 07 Aug 2003 21:28:59 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19kw3x-0006mo-33; Thu, 07 Aug 2003 21:29:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19kw32-0006mN-SE for rddp@optimus.ietf.org; Thu, 07 Aug 2003 21:28:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA11307 for ; Thu, 7 Aug 2003 21:27:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19kw30-0004LK-00 for rddp@ietf.org; Thu, 07 Aug 2003 21:28:02 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19kw2z-0004LA-00 for rddp@ietf.org; Thu, 07 Aug 2003 21:28:01 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 25EFE2453B6; Thu, 7 Aug 2003 21:27:33 -0400 (EDT) Date: Thu, 7 Aug 2003 20:27:16 -0500 Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: , To: Mike Ko From: Caitlin Bestler In-Reply-To: Message-Id: <72281D4D-C93F-11D7-ABA1-003065D48EE0@asomi.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Subject: [rddp] Re: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Thursday, August 7, 2003, at 12:52 PM, Mike Ko wrote: > Caitlin, since MaxCmdSN is used by the target to limit the number of > commands it can receive from the initiator, your suggestion would work > for > limiting the number of "fringe" messages the initiator can send to the > target. However, for "fringe" messages from the target to the > initiator, > such as the Aysnchronous Message, the target cannot simply raise > MaxCmdSN > and expect the initiator to increase the number of receive buffers for > "fringe" messages correspondingly. > > Mallikarjun is correct in pointing out that some form of new wire > protocol > for flow control is required if we need to flow control the messages > not > regulated by CmdSN. > The target may always send untagged messages when they are a response to an initiator request. Additionally the target may have N extra outstanding untagged messages, where N is a protocol constant or negotiated for the session. A non-responding target message is "outstanding" until one of the following occurs: The initiator responds to a "Nop Out Request". The initiator sends a command with CmdSN that was only enabled after the non-responding target message. The fact that a prior non-responding target untagged message is no longer "outstanding" merely means that whatever buffers are allocated for this purpose have been replaced. It does not guarantee that the initiator has completed any associated processing, that logic is left totally to the ULP. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Aug 8 17:33:28 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA26786 for ; Fri, 8 Aug 2003 17:33:28 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19lErA-00018F-7R for rddp-archive@odin.ietf.org; Fri, 08 Aug 2003 17:33:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h78LX4JG004350 for rddp-archive@odin.ietf.org; Fri, 8 Aug 2003 17:33:04 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19lErA-000185-4B for rddp-web-archive@optimus.ietf.org; Fri, 08 Aug 2003 17:33:04 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA26779 for ; Fri, 8 Aug 2003 17:32:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19lEr7-0004OX-00 for rddp-web-archive@ietf.org; Fri, 08 Aug 2003 17:33:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19lEr6-0004OU-00 for rddp-web-archive@ietf.org; Fri, 08 Aug 2003 17:33:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19lEr7-00017Y-Hr; Fri, 08 Aug 2003 17:33:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19lEqn-000178-Ms for rddp@optimus.ietf.org; Fri, 08 Aug 2003 17:32:42 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA26771 for ; Fri, 8 Aug 2003 17:32:35 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19lEql-0004OL-00 for rddp@ietf.org; Fri, 08 Aug 2003 17:32:39 -0400 Received: from e31.co.us.ibm.com ([32.97.110.129]) by ietf-mx with esmtp (Exim 4.12) id 19lEqk-0004OF-00 for rddp@ietf.org; Fri, 08 Aug 2003 17:32:38 -0400 Received: from westrelay04.boulder.ibm.com (westrelay04.boulder.ibm.com [9.17.193.32]) by e31.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h78LW5RZ265002; Fri, 8 Aug 2003 17:32:05 -0400 Received: from d03nm694.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay04.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h78LW3ve061600; Fri, 8 Aug 2003 15:32:04 -0600 Importance: Normal MIME-Version: 1.0 To: Caitlin Bestler Cc: , X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: Mike Ko Date: Fri, 8 Aug 2003 14:32:02 -0700 X-MIMETrack: Serialize by Router on D03NM694/03/M/IBM(Release 6.0.1 [IBM]|June 10, 2003) at 08/08/2003 15:32:04, Serialize complete at 08/08/2003 15:32:04 Content-Type: text/plain; charset="us-ascii" Subject: [rddp] Re: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Since the Asynchronous Message is associated with a StatSN, instead of waiting for the initiator to send "a command with CmdSN that was only enabled after the non-responding target message", another possibility is to wait for the initiator to send any PDU containing the ExtStatSN field that acknowledges that the particular StatSN has been received. Mike Ko IBM Almaden Research mako@almaden.ibm.com To: Mike Ko cc: , Subject: Re: iSCSI/iWARP drafts and flow control On Thursday, August 7, 2003, at 12:52 PM, Mike Ko wrote: > Caitlin, since MaxCmdSN is used by the target to limit the number of > commands it can receive from the initiator, your suggestion would work > for > limiting the number of "fringe" messages the initiator can send to the > target. However, for "fringe" messages from the target to the > initiator, > such as the Aysnchronous Message, the target cannot simply raise > MaxCmdSN > and expect the initiator to increase the number of receive buffers for > "fringe" messages correspondingly. The target may always send untagged messages when they are a response to an initiator request. Additionally the target may have N extra outstanding untagged messages, where N is a protocol constant or negotiated for the session. A non-responding target message is "outstanding" until one of the following occurs: The initiator responds to a "Nop Out Request". The initiator sends a command with CmdSN that was only enabled after the non-responding target message. The fact that a prior non-responding target untagged message is no longer "outstanding" merely means that whatever buffers are allocated for this purpose have been replaced. It does not guarantee that the initiator has completed any associated processing, that logic is left totally to the ULP. _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Sat Aug 9 14:45:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA03452 for ; Sat, 9 Aug 2003 14:45:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19lYi7-0005O2-W1 for rddp-archive@odin.ietf.org; Sat, 09 Aug 2003 14:45:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h79Ij39H020702 for rddp-archive@odin.ietf.org; Sat, 9 Aug 2003 14:45:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19lYi7-0005Np-Pw for rddp-web-archive@optimus.ietf.org; Sat, 09 Aug 2003 14:45:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA03446 for ; Sat, 9 Aug 2003 14:44:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19lYi5-0002SF-00 for rddp-web-archive@ietf.org; Sat, 09 Aug 2003 14:45:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19lYi4-0002SB-00 for rddp-web-archive@ietf.org; Sat, 09 Aug 2003 14:45:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19lYi4-0005NU-NZ; Sat, 09 Aug 2003 14:45:00 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19lYhj-0005ND-9g for rddp@optimus.ietf.org; Sat, 09 Aug 2003 14:44:39 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA03438 for ; Sat, 9 Aug 2003 14:44:33 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19lYhg-0002S5-00 for rddp@ietf.org; Sat, 09 Aug 2003 14:44:36 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19lYhg-0002RK-00 for rddp@ietf.org; Sat, 09 Aug 2003 14:44:36 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 3A7CE244C53; Sat, 9 Aug 2003 14:44:12 -0400 (EDT) Date: Sat, 9 Aug 2003 13:43:55 -0500 Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: , To: Mike Ko From: Caitlin Bestler In-Reply-To: Message-Id: <6E2BBF37-CA99-11D7-9E17-003065D48EE0@asomi.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Subject: [rddp] Re: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Friday, August 8, 2003, at 04:32 PM, Mike Ko wrote: > Since the Asynchronous Message is associated with a StatSN, instead of > waiting for the initiator to send "a command with CmdSN that was only > enabled after the non-responding target message", another possibility > is > to wait for the initiator to send any PDU containing the ExtStatSN > field > that acknowledges that the particular StatSN has been received. Even better. I am not favoring any specific mechanism, just saying that mechanisms exist -- and that they do not have to be an explicit "protocol". Note that none of the algorithms described, including your suggestion, require the receiver to calculate or issue credits. They merely have to replenish buffers in manner consistent with the rules. They can also be sloppy about it if they have slop in their buffers. The rules suggested shift the burden of flow control to the sender, where it should be. Rule enforcement at the sending side can also be quite lax if other implementation specific rules limit the number of untagged messages that could be sent. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Aug 15 17:14:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14076 for ; Fri, 15 Aug 2003 17:14:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19nltb-0004It-K4 for rddp-archive@odin.ietf.org; Fri, 15 Aug 2003 17:14:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h7FLE3pA016537 for rddp-archive@odin.ietf.org; Fri, 15 Aug 2003 17:14:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19nltb-0004Ie-EW for rddp-web-archive@optimus.ietf.org; Fri, 15 Aug 2003 17:14:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14063 for ; Fri, 15 Aug 2003 17:13:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19nltZ-0006FK-00 for rddp-web-archive@ietf.org; Fri, 15 Aug 2003 17:14:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19nltY-0006FG-00 for rddp-web-archive@ietf.org; Fri, 15 Aug 2003 17:14:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19nltZ-0004IQ-AW; Fri, 15 Aug 2003 17:14:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19nltP-0004CT-68 for rddp@optimus.ietf.org; Fri, 15 Aug 2003 17:13:53 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id RAA14058 for ; Fri, 15 Aug 2003 17:13:45 -0400 (EDT) From: wrstuden@wasabisystems.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19nltM-0006FC-00 for rddp@ietf.org; Fri, 15 Aug 2003 17:13:48 -0400 Received: from [151.199.66.145] (helo=mononoke.wasabisystems.com) by ietf-mx with esmtp (Exim 4.12) id 19nltM-0006F8-00 for rddp@ietf.org; Fri, 15 Aug 2003 17:13:48 -0400 Received: by mononoke.wasabisystems.com (Postfix, from userid 1021) id 5375D40139; Fri, 15 Aug 2003 17:13:43 -0400 (EDT) Date: Fri, 15 Aug 2003 14:12:00 -0700 (PDT) X-X-Sender: wrstuden@neverwinter.home-net.icnt.net To: Caitlin Bestler Cc: Mike Ko , ips@ece.cmu.edu, rddp@ietf.org Subject: Re: [rddp] Re: iSCSI/iWARP drafts and flow control In-Reply-To: <6E2BBF37-CA99-11D7-9E17-003065D48EE0@asomi.com> Message-ID: References: <6E2BBF37-CA99-11D7-9E17-003065D48EE0@asomi.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , On Sat, 9 Aug 2003, Caitlin Bestler wrote: > The rules suggested shift the burden of flow control > to the sender, where it should be. Rule enforcement > at the sending side can also be quite lax if other > implementation specific rules limit the number of > untagged messages that could be sent. Sorry for my delay in responding. One of the things I've been wondering through this whole thread is can't we expand iSCSI negotiation to handle these parameters (like how many async messages are ok, how many immediate commands, etc.)? That way, to the extent that iSCSI/iWARP needs new parameters, we can decide on them out easily. Maybe it's been in everyone's thoughts, but the thread had been talking about needing new limits and talking like it's hard to do. Can't we just negotiate them? Take care, Bill _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Aug 15 18:15:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA16231 for ; Fri, 15 Aug 2003 18:15:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19nmqd-0007ah-Ui for rddp-archive@odin.ietf.org; Fri, 15 Aug 2003 18:15:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h7FMF3IZ029175 for rddp-archive@odin.ietf.org; Fri, 15 Aug 2003 18:15:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19nmqd-0007aU-Rc for rddp-web-archive@optimus.ietf.org; Fri, 15 Aug 2003 18:15:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA16190 for ; Fri, 15 Aug 2003 18:14:57 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19nmqa-0006W7-00 for rddp-web-archive@ietf.org; Fri, 15 Aug 2003 18:15:00 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19nmqa-0006W4-00 for rddp-web-archive@ietf.org; Fri, 15 Aug 2003 18:15:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19nmqb-0007a4-AT; Fri, 15 Aug 2003 18:15:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19nmpu-0007Zf-W5 for rddp@optimus.ietf.org; Fri, 15 Aug 2003 18:14:19 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id SAA16115 for ; Fri, 15 Aug 2003 18:14:12 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19nmps-0006Vt-00 for rddp@ietf.org; Fri, 15 Aug 2003 18:14:16 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19nmpr-0006Vc-00 for rddp@ietf.org; Fri, 15 Aug 2003 18:14:15 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 627E3245D86; Fri, 15 Aug 2003 18:13:55 -0400 (EDT) Date: Fri, 15 Aug 2003 17:13:37 -0500 Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: Mike Ko , ips@ece.cmu.edu, rddp@ietf.org To: wrstuden@wasabisystems.com From: Caitlin Bestler In-Reply-To: Message-Id: Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Subject: [rddp] Re: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Friday, August 15, 2003, at 04:12 PM, wrstuden@wasabisystems.com wrote: > Sorry for my delay in responding. > > One of the things I've been wondering through this whole thread is > can't > we expand iSCSI negotiation to handle these parameters (like how many > async messages are ok, how many immediate commands, etc.)? That way, to > the extent that iSCSI/iWARP needs new parameters, we can decide on them > out easily. > > Maybe it's been in everyone's thoughts, but the thread had been talking > about needing new limits and talking like it's hard to do. Can't we > just > negotiate them? > > Take care, > > Bill Bill, There are two separate issues. The first is establishing what the credit limits are. That could be negotiated on a per session basis, or it could be established by the draft as a simple rule. You are correct that negotiating is not terribly complex. Therefore, if there is *any* benefit to varying those limits on a per-session basis, then negotiations should be allowed. So far, however, I do not believe that anyone has identified a specific need for per-session negotiation of these credits. The second issue is the mechanism by which credits are replenished. The iSER draft authors were opposed to creation of additional messages to exchange credits. My contention is that under virtually all circumstances, credit replenishment can be inferred from existing message exchanges. Under worst case circumstances a "NOP" might have to be sent that would not have otherwise been required. No additional message types are required. I believe the last few prior exchanges may have established that tracking credits is not as onerous as the draft authors had originally thought. But it is on that issue, not the overhead of negotiation during setup, that was their concern. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Aug 15 21:05:29 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA19692 for ; Fri, 15 Aug 2003 21:05:29 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19npV9-0006eF-Li for rddp-archive@odin.ietf.org; Fri, 15 Aug 2003 21:05:03 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h7G153wW025549 for rddp-archive@odin.ietf.org; Fri, 15 Aug 2003 21:05:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19npV9-0006e0-IO for rddp-web-archive@optimus.ietf.org; Fri, 15 Aug 2003 21:05:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA19684 for ; Fri, 15 Aug 2003 21:04:58 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19npV7-0007U3-00 for rddp-web-archive@ietf.org; Fri, 15 Aug 2003 21:05:01 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19npV6-0007U0-00 for rddp-web-archive@ietf.org; Fri, 15 Aug 2003 21:05:00 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19npV7-0006dR-9h; Fri, 15 Aug 2003 21:05:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19npUw-0006cv-Mk for rddp@optimus.ietf.org; Fri, 15 Aug 2003 21:04:50 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id VAA19676 for ; Fri, 15 Aug 2003 21:04:46 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19npUu-0007Th-00 for rddp@ietf.org; Fri, 15 Aug 2003 21:04:48 -0400 Received: from e35.co.us.ibm.com ([32.97.110.133]) by ietf-mx with esmtp (Exim 4.12) id 19npUt-0007TV-00 for rddp@ietf.org; Fri, 15 Aug 2003 21:04:47 -0400 Received: from westrelay02.boulder.ibm.com (westrelay02.boulder.ibm.com [9.17.195.11]) by e35.co.us.ibm.com (8.12.9/8.12.2) with ESMTP id h7G14Bs5481566; Fri, 15 Aug 2003 21:04:11 -0400 Received: from d03nm694.boulder.ibm.com (d03av02.boulder.ibm.com [9.17.193.82]) by westrelay02.boulder.ibm.com (8.12.9/NCO/VER6.5) with ESMTP id h7G14A0b297542; Fri, 15 Aug 2003 19:04:11 -0600 Importance: Normal MIME-Version: 1.0 To: Caitlin Bestler Cc: wrstuden@wasabisystems.com, ips@ece.cmu.edu, rddp@ietf.org X-Mailer: Lotus Notes Release 5.0.7 March 21, 2001 Message-ID: From: Mike Ko Date: Fri, 15 Aug 2003 18:04:08 -0700 X-MIMETrack: Serialize by Router on D03NM694/03/M/IBM(Release 6.0.1 [IBM]|June 10, 2003) at 08/15/2003 19:04:11, Serialize complete at 08/15/2003 19:04:11 Content-Type: text/plain; charset="us-ascii" Subject: [rddp] Re: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , As I indicated to Caitlin in an earlier e-mail, since each iSCSI control-type PDU from the target to the initiator is associated with a StatSN, the ExpStatSN from the initiator can be used to advance the "credit" at the target. The question is for iSCSI control-type PDUs from the initiator to the target, what is the mechanism for the target to advance the "credit" at the initiator? For PDUs where the Immediate flag is not set, the CmdSN and MaxCmdSN mechanism in iSCSI are already available and the new "credit" scheme is not necessary. The unsolicited SCSI Data-out PDUs are covered by FirstBurstLength and again the new "credit" scheme is not needed. The "credit" scheme is needed for PDUs where the Immediate flag is set or the SNACK Request (note that SNACKs are not needed in iSER). For these PDUs, the target may not be able to advance the "credit" at the initiator individually. One possibility is to wait for the initiator to issue an iSCSI control-type PDU with a new CmdSN but without the Immediate flag, and if the target responds with an iSCSI control-type PDU with ExpCmdSN set to CmdSN + 1, then "credit" at the initiator is restored to its full negotiated (or by definition) value. As Caitlin pointed out, since no one has yet identified a specific need for per-session negotiation of these credits, a simple rule enumerated in the draft may be sufficient. Mike Ko IBM Almaden Research mako@almaden.ibm.com To: wrstuden@wasabisystems.com cc: Mike Ko , ips@ece.cmu.edu, rddp@ietf.org Subject: Re: iSCSI/iWARP drafts and flow control On Friday, August 15, 2003, at 04:12 PM, wrstuden@wasabisystems.com wrote: > Sorry for my delay in responding. > > One of the things I've been wondering through this whole thread is > can't > we expand iSCSI negotiation to handle these parameters (like how many > async messages are ok, how many immediate commands, etc.)? That way, to > the extent that iSCSI/iWARP needs new parameters, we can decide on them > out easily. > > Maybe it's been in everyone's thoughts, but the thread had been talking > about needing new limits and talking like it's hard to do. Can't we > just > negotiate them? > > Take care, > > Bill Bill, There are two separate issues. The first is establishing what the credit limits are. That could be negotiated on a per session basis, or it could be established by the draft as a simple rule. You are correct that negotiating is not terribly complex. Therefore, if there is *any* benefit to varying those limits on a per-session basis, then negotiations should be allowed. So far, however, I do not believe that anyone has identified a specific need for per-session negotiation of these credits. The second issue is the mechanism by which credits are replenished. The iSER draft authors were opposed to creation of additional messages to exchange credits. My contention is that under virtually all circumstances, credit replenishment can be inferred from existing message exchanges. Under worst case circumstances a "NOP" might have to be sent that would not have otherwise been required. No additional message types are required. I believe the last few prior exchanges may have established that tracking credits is not as onerous as the draft authors had originally thought. But it is on that issue, not the overhead of negotiation during setup, that was their concern. Caitlin Bestler - cait@asomi.com - http://asomi.com/ _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Aug 15 22:24:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA20719 for ; Fri, 15 Aug 2003 22:24:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19nqjd-0000vZ-Js for rddp-archive@odin.ietf.org; Fri, 15 Aug 2003 22:24:05 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h7G2O5NL003559 for rddp-archive@odin.ietf.org; Fri, 15 Aug 2003 22:24:05 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19nqjd-0000vK-G0 for rddp-web-archive@optimus.ietf.org; Fri, 15 Aug 2003 22:24:05 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA20693 for ; Fri, 15 Aug 2003 22:23:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19nqja-0000Fu-00 for rddp-web-archive@ietf.org; Fri, 15 Aug 2003 22:24:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19nqjZ-0000Fr-00 for rddp-web-archive@ietf.org; Fri, 15 Aug 2003 22:24:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19nqjZ-0000uz-2Y; Fri, 15 Aug 2003 22:24:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19nqiw-0000uo-Gd for rddp@optimus.ietf.org; Fri, 15 Aug 2003 22:23:22 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id WAA20684 for ; Fri, 15 Aug 2003 22:23:16 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19nqit-0000FP-00 for rddp@ietf.org; Fri, 15 Aug 2003 22:23:19 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19nqis-0000Eq-00 for rddp@ietf.org; Fri, 15 Aug 2003 22:23:18 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id 5FE69245DFD; Fri, 15 Aug 2003 22:23:03 -0400 (EDT) Date: Fri, 15 Aug 2003 21:22:45 -0500 Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: wrstuden@wasabisystems.com, ips@ece.cmu.edu, rddp@ietf.org To: Mike Ko From: Caitlin Bestler In-Reply-To: Message-Id: <85AFC7C8-CF90-11D7-99E9-003065D48EE0@asomi.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Subject: [rddp] Re: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Friday, August 15, 2003, at 08:04 PM, Mike Ko wrote: > As I indicated to Caitlin in an earlier e-mail, since each iSCSI > control-type PDU from the target to the initiator is associated with a > StatSN, the ExpStatSN from the initiator can be used to advance the > "credit" at the target. Agreed. > > The question is for iSCSI control-type PDUs from the initiator to the > target, what is the mechanism for the target to advance the "credit" at > the initiator? For PDUs where the Immediate flag is not set, the CmdSN > and MaxCmdSN mechanism in iSCSI are already available and the new > "credit" > scheme is not necessary. The unsolicited SCSI Data-out PDUs are > covered > by FirstBurstLength and again the new "credit" scheme is not needed. > The > "credit" scheme is needed for PDUs where the Immediate flag is set or > the > SNACK Request (note that SNACKs are not needed in iSER). For these > PDUs, > the target may not be able to advance the "credit" at the initiator > individually. One possibility is to wait for the initiator to issue an > iSCSI control-type PDU with a new CmdSN but without the Immediate flag, > and if the target responds with an iSCSI control-type PDU with ExpCmdSN > set to CmdSN + 1, then "credit" at the initiator is restored to its > full > negotiated (or by definition) value. My proposal is to apply a "leaky bucket" algorithm. "Extra" credits are granted for each CmdSN credit granted, up to a maximum. Each extra send decrements a fixed amount from the credits. In a pinch, Nop commands can be used to advance the CmdSN. It allows for bursts, and the credit-scoring can be done entirely when sending a message that needs an "extra" credit. > > As Caitlin pointed out, since no one has yet identified a specific need > for per-session negotiation of these credits, a simple rule enumerated > in > the draft may be sufficient. > Well, their negotiated credits in either case. It's just whether they are negotiated on a per-session basis by target and initiator, or negotiated on a mailing list on a per-protocol basis. If the volume of the mailing list negotiation gets too high, everyone will quickly agree to the per-session negotiations. ;-) _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Aug 18 08:27:31 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA17470 for ; Mon, 18 Aug 2003 08:27:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19oj6F-0003xg-N6 for rddp-archive@odin.ietf.org; Mon, 18 Aug 2003 08:27:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h7ICR3Zh015222 for rddp-archive@odin.ietf.org; Mon, 18 Aug 2003 08:27:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19oj6F-0003xR-I9 for rddp-web-archive@optimus.ietf.org; Mon, 18 Aug 2003 08:27:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA17467 for ; Mon, 18 Aug 2003 08:26:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19oj6E-0006sn-00 for rddp-web-archive@ietf.org; Mon, 18 Aug 2003 08:27:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19oj6D-0006sj-00 for rddp-web-archive@ietf.org; Mon, 18 Aug 2003 08:27:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19oj6D-0003x5-7H; Mon, 18 Aug 2003 08:27:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19oj5v-0003wq-2u for rddp@optimus.ietf.org; Mon, 18 Aug 2003 08:26:45 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id IAA17451 for ; Mon, 18 Aug 2003 08:26:39 -0400 (EDT) From: wrstuden@wasabisystems.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19oj5t-0006sg-00 for rddp@ietf.org; Mon, 18 Aug 2003 08:26:41 -0400 Received: from [151.199.66.145] (helo=mononoke.wasabisystems.com) by ietf-mx with esmtp (Exim 4.12) id 19oj5t-0006sc-00 for rddp@ietf.org; Mon, 18 Aug 2003 08:26:41 -0400 Received: by mononoke.wasabisystems.com (Postfix, from userid 1021) id 6264740134; Mon, 18 Aug 2003 08:26:40 -0400 (EDT) Date: Mon, 18 Aug 2003 05:24:53 -0700 (PDT) X-X-Sender: wrstuden@neverwinter To: Caitlin Bestler Cc: Mike Ko , ips@ece.cmu.edu, rddp@ietf.org In-Reply-To: <85AFC7C8-CF90-11D7-99E9-003065D48EE0@asomi.com> Message-ID: References: <85AFC7C8-CF90-11D7-99E9-003065D48EE0@asomi.com> MIME-Version: 1.0 Content-Type: TEXT/PLAIN; charset=US-ASCII Subject: [rddp] Re: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , On Fri, 15 Aug 2003, Caitlin Bestler wrote: > On Friday, August 15, 2003, at 08:04 PM, Mike Ko wrote: > > > As Caitlin pointed out, since no one has yet identified a specific > > need for per-session negotiation of these credits, a simple rule > > enumerated in the draft may be sufficient. > > Well, their negotiated credits in either case. It's just whether > they are negotiated on a per-session basis by target and > initiator, or negotiated on a mailing list on a per-protocol > basis. If the volume of the mailing list negotiation gets > too high, everyone will quickly agree to the per-session > negotiations. ;-) I'm not saying we can't just pick a number. But if I understand things, one advantage of per-session negotiation might be that a target with fixed buffering could offer different command windows to initiators that wanted different numbers of immediate commands available. For an initiator that is more comfortable advancing CmdSN (is happier with a lower number of credits per CmdSN), a target could divy up its buffers to allow a wider command window. Take care, Bill _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Mon Aug 18 11:36:30 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA22889 for ; Mon, 18 Aug 2003 11:36:30 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19om39-0001Tl-NK for rddp-archive@odin.ietf.org; Mon, 18 Aug 2003 11:36:04 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h7IFa38P005670 for rddp-archive@odin.ietf.org; Mon, 18 Aug 2003 11:36:03 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19om39-0001TH-HH for rddp-web-archive@optimus.ietf.org; Mon, 18 Aug 2003 11:36:03 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA22878 for ; Mon, 18 Aug 2003 11:35:59 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19om38-00003T-00 for rddp-web-archive@ietf.org; Mon, 18 Aug 2003 11:36:02 -0400 Received: from ietf.org ([132.151.1.19] helo=optimus.ietf.org) by ietf-mx with esmtp (Exim 4.12) id 19om37-00003Q-00 for rddp-web-archive@ietf.org; Mon, 18 Aug 2003 11:36:01 -0400 Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19om37-0001SZ-HD; Mon, 18 Aug 2003 11:36:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19om2S-0001Rb-NI for rddp@optimus.ietf.org; Mon, 18 Aug 2003 11:35:20 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id LAA22870 for ; Mon, 18 Aug 2003 11:35:16 -0400 (EDT) Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19om2R-00003L-00 for rddp@ietf.org; Mon, 18 Aug 2003 11:35:19 -0400 Received: from thebe.your-site.com ([140.186.45.26]) by ietf-mx with esmtp (Exim 4.12) id 19om2R-00003H-00 for rddp@ietf.org; Mon, 18 Aug 2003 11:35:19 -0400 Received: from asomi.com (64-144-5-25.client.dsl.net [64.144.5.25]) by thebe.your-site.com (Postfix) with ESMTP id AB734244B71; Mon, 18 Aug 2003 11:35:03 -0400 (EDT) Date: Mon, 18 Aug 2003 10:34:45 -0500 Content-Type: text/plain; charset=US-ASCII; format=flowed Mime-Version: 1.0 (Apple Message framework v552) Cc: ips@ece.cmu.edu, rddp@ietf.org To: wrstuden@wasabisystems.com From: Caitlin Bestler In-Reply-To: Message-Id: <7E714293-D191-11D7-B04A-003065D48EE0@asomi.com> Content-Transfer-Encoding: 7bit X-Mailer: Apple Mail (2.552) Content-Transfer-Encoding: 7bit Subject: [rddp] Re: iSCSI/iWARP drafts and flow control Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Content-Transfer-Encoding: 7bit Content-Transfer-Encoding: 7bit On Monday, August 18, 2003, at 07:24 AM, wrstuden@wasabisystems.com wrote: > I'm not saying we can't just pick a number. But if I understand things, > one advantage of per-session negotiation might be that a target with > fixed > buffering could offer different command windows to initiators that > wanted > different numbers of immediate commands available. For an initiator > that > is more comfortable advancing CmdSN (is happier with a lower number of > credits per CmdSN), a target could divy up its buffers to allow a wider > command window. The cost of per-session negotiation is minor, so I suspect that almost any solid example of an application or scenario that would benefit from a non-standard number of credits would be convincing. But given that the flow control is imposed on the sending side, and that these are rare packets, it isn't that easy to come up with such a scenario. I'm not opposed to the idea, but if a parameter is going to be listed as negotiable in the draft there should be at least a paragraph hinting at why you might want to vary it (even if it was only on the mailing list and doesn't make it into the draft). _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp From exim@www1.ietf.org Fri Aug 22 14:01:58 2003 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA10694 for ; Fri, 22 Aug 2003 14:01:58 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19qGE9-0008OX-6e for rddp-archive@odin.ietf.org; Fri, 22 Aug 2003 14:01:33 -0400 Received: (from exim@localhost) by www1.ietf.org (8.12.8/8.12.8/Submit) id h7MI1XNA032263 for rddp-archive@odin.ietf.org; Fri, 22 Aug 2003 14:01:33 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19qGE9-0008OI-2N for rddp-web-archive@optimus.ietf.org; Fri, 22 Aug 2003 14:01:33 -0400 Received: from optimus.ietf.org (ietf.org [132.151.1.19] (may be forged)) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA10675 for ; Fri, 22 Aug 2003 14:01:27 -0400 (EDT) Received: from localhost.localdomain ([127.0.0.1] helo=www1.ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19qGDd-0008Lx-V5; Fri, 22 Aug 2003 14:01:01 -0400 Received: from odin.ietf.org ([132.151.1.176] helo=ietf.org) by optimus.ietf.org with esmtp (Exim 4.20) id 19qGCu-0008Kz-5l for rddp@optimus.ietf.org; Fri, 22 Aug 2003 14:00:16 -0400 Received: from ietf-mx (ietf-mx.ietf.org [132.151.6.1]) by ietf.org (8.9.1a/8.9.1a) with ESMTP id OAA10608 for ; Fri, 22 Aug 2003 14:00:11 -0400 (EDT) From: Black_David@emc.com Received: from ietf-mx ([132.151.6.1]) by ietf-mx with esmtp (Exim 4.12) id 19qGCr-0007hi-00 for rddp@ietf.org; Fri, 22 Aug 2003 14:00:13 -0400 Received: from mxic2.corp.emc.com ([128.221.31.40]) by ietf-mx with esmtp (Exim 4.12) id 19qGCr-0007hU-00 for rddp@ietf.org; Fri, 22 Aug 2003 14:00:13 -0400 Received: by mxic2.corp.emc.com with Internet Mail Service (5.5.2653.19) id ; Fri, 22 Aug 2003 13:59:43 -0400 Message-ID: To: rddp@ietf.org Date: Fri, 22 Aug 2003 13:59:38 -0400 MIME-Version: 1.0 X-Mailer: Internet Mail Service (5.5.2653.19) Content-Type: text/plain Subject: [rddp] Delay in posts from non-subscribers Sender: rddp-admin@ietf.org Errors-To: rddp-admin@ietf.org X-BeenThere: rddp@ietf.org X-Mailman-Version: 2.0.12 Precedence: bulk List-Unsubscribe: , List-Id: IETF Remote Direct Data Placement (rddp) WG List-Post: List-Help: List-Subscribe: , Due to the volume of spam being generated by the latest Sobig.F worm, it is no longer possible for me to review messages sent to the RDDP mailing list by non-subscribers in a timely fashion. Normally, I review messages from non-subscribers more or less as they come in and approve ones that are not spam. This will now take place at most once a day. If you are subscribed to the list, you are not affected by this - your messages will continue to go through immediately and automatically. If you are not a list subscriber and send email to the list, it'll probably take until sometime the next day (or over the weekend until Monday) for your post to be examined and approved. I apologize for this delay, but there haven't been many recent posts that would have been hit by this restriction. Anyone who wants to post to the rddp list really should subscribe. Also, if anyone knows how to get through to Road Runner, the current bombardment of spam to the RDDP list is coming from one Road Runner host: (rrcs-se-24-73-64-88.biz.rr.com [24.73.64.88]) Thanks, --David ---------------------------------------------------- David L. Black, Senior Technologist EMC Corporation, 176 South St., Hopkinton, MA 01748 +1 (508) 293-7953 FAX: +1 (508) 293-7786 black_david@emc.com Mobile: +1 (978) 394-7754 ---------------------------------------------------- _______________________________________________ rddp mailing list rddp@ietf.org https://www1.ietf.org/mailman/listinfo/rddp